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1.0  Executive  Summary 

1.1  Objective 

The  U.S.  Navy  has  the  responsibility  to  develop  a  comprehensive,  worldwide  Maritime 
Domain  Awareness  (MDA)  capability.  The  Deputy  Under  Secretary  of  the  Navy 
(DUSN)  has  appointed  the  Naval  Postgraduate  School  (NPS)  to  assess  the  overall 
capabilities  and  needs  of  MDA.  The  objective  of  this  document  is  to  report  the  progress 
made  in  FY08  to  improve  the  effectiveness  of  the  warfighter  in  MDA  missions. 

This  report  focuses  primarily  on  the  set  of  MDA  systems  that  Program  Executive 
Office  (PEO),  Command,  Control,  Computers,  Communications,  and  Intelligence  (C4I) 
has  designated  Spiral- 1 .  The  assessment  is  based  on  data  from  numerous  EY08  tests, 
experiments,  and  studies  of  MDA  systems  directed  by  PEO  C4I  and  conducted  by 
Operational  Test  &  Evaluation  Eorce  (OPTEVEOR  )  in  EY08  or  by  NPS.  Specifically, 
this  report  aggregates  and  analyzes  observations,  surveys,  chat  logs,  and  interviews  from 
six  experimental  and  practical  exercises  that  use  MDA  Spiral- 1  technologies,  and  from 
other  events  in  which  those  technologies  were  discussed  by  operational  personnel.  To 
these  data,  we  have  applied  a  comprehensive  assessment  structure  and  method  that  have 
been  developed  and  validated  in  the  Trident  Warrior  exercises  that  began  in  2002.  The 
assessment  structure  is  customized  to  MDA,  and  it  can  be  used  throughout  the  MDA 
program  to  evaluate  technologies,  organizations,  processes  and  other  enabling 
components  of  the  MDA  solution.  Our  findings  and  recommendations  are  presented 
using  this  structure.  (The  specifics  of  this  structure  are  explained  in  Section  3,  below.) 

Note  that  raw  data  from  the  OPTEVEOR  Quick  Reaction  Assessment  (QRA) 
(EAIRGAME)  event  is  not  available,  and  is  therefore  not  addressed  in  this  report. 

1 .2  MDA  Program 

The  MDA  program  spans  many  regions,  technologies,  and  policies. 

MDA  was  triggered  by  a  Presidential  Initiative  (White  House,  1998).  This,  in 
turn,  stimulated  a  requirement  for  a  QRA  to  be  conducted  for  MDA  Spiral- 1  Acceleration 
leave  behind  capabilities. 

A  Secretary  of  the  Navy  (SECNAV)  project  was  promulgated  and  is  managed  by 
PEO,  C4I  to  increase  the  United  States  maritime  security  through  MDA.  This  MDA 
project  will  develop  an  enhanced  capability  to  identify  threats  in  the  Maritime  Domain  as 
early  and  as  distant  from  our  shores  as  possible.  It  will  do  so  by  integrating  intelligence, 
surveillance,  observation,  and  navigation  systems  into  a  common  operating  picture  (COP) 
accessible  throughout  the  U.S.  Government. 

The  MDA  Science  and  Technology  (S&T)  Prototype  develops  technological 
systems  to  support  MDA.  These  systems,  collectively,  are  designated  MDA  Spiral- 1.  The 
effort  is  in  the  formative  stage.  Thus,  the  prototype  capabilities  have  broadly  stated 
capability  thresholds  and  objectives. 
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1.3  Analysis  Method 


The  Maritime  Domain  Awareness  (MDA)  program  has  conducted  a  series  of  tests  of 
Spiral- 1  systems.  PEO  C4I  has  received  report  results  from  these  individual  tests.  This 
report  looks  across  these  tests  and  other  events  conducted  by  the  Naval  Postgraduate 
School  (NPS)  to  assess  overall  MDA  capabilities.  The  venues  from  which  we  have 
drawn  data  for  this  assessment  are; 

•  NPS  Site  Visits  for  workflow  development  &  a  Process  Engineering  Workshop 
(PEW) 

•  Test  Risk  Reduction  Eimited  Objective  Experiment  (TRRLOE) 

•  Visit  Board  Search  and  Seizure  (VBSS)  School 

•  Trident  Warrior  08 

•  Empire  Challenge  08 

•  EAIRGAME 

The  data  from  these  venues  were  largely  qualitative  -  including  survey  comments, 
chat  logs,  data  collector’s  observations,  and  interview  notes.  Some  quantitative  survey 
data  were  also  available.  These  data  were  assigned  to  categories  of  the  assessment 
framework,  which  consisted  of  a  hierarchy  of  assessment  areas  concerning:  System 
Performance,  Operations  Performance,  Organization/Guidance,  and  System 
Supportability  and  Readiness.  (None  of  these  data  directly  addressed  a  fifth  assessment 
area:  Warfighter  Performance).  Each  item  in  each  category  received  a  rating  indicating 
that  it  described  a  strength,  a  concern,  or  a  deficiency  in  an  MDA  system,  suite  of 
systems,  or  deployment  concept  for  systems 

1 .4  Summary  of  F indings 

Most  of  the  data  analyzed  here  described  strengths  or  concerns  of  MDA  systems.  Eew 
users  identified  deficiencies  in  these  systems.  The  scores  indicate  that  all  of  the  assessed 
technologies  increased  the  warfighter’s  effectiveness  to  some  degree,  yet  there  was  little 
information  about  the  level  of  enhancement  of  MDA  capabilities  provided  by  the  suite  of 
Spiral- 1  systems  due  to  the  lack  of  data  on  baseline  levels  of  performance. 

Specifically,  in  the  quantitative  assessment,  areas  of  greatest  concern  (lowest 
average  scores)  were: 

>  Operations  Performance  for  EMIO* 

>  Several  sub-areas  of  Organization/Guidance  concerning  MDA 
Compatibility  (specifically,  alignment  of  MDA  activity  with  ONI) 

>  Guidance  (especially  the  need  for  MDA  CONOPS,  TTPs,  and  SOPs) 

>  Agreements  (concerning  data  sharing) 

>  System  Management 

>  Security 

>  System  Supportability 

>  Readiness 
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*Assessment  for  EMIO  was  obtained  through  an  actual  operational 
experiment,  not  a  system  test.  This  differs  from  the  rest  of  the  results  which  were 
obtained  from  structured  laboratory  events. 

The  most  highly  rated  sub-areas  were: 

>  Operations  Performance  concerning  Knowledge  Processes  for  vessel  of 
interest  (VOI  )  development  and  tracking 

>  Organization/Guidance  concerning  Maritime  Headquarters  with  Maritime 
Operations  Center  (MHQ/MOC)  Compatibility  with  MDA 

>  System  Performance  concerning  the  Operations  Support  provided  by  these 
systems. 

Finer  categorization,  a  synopsis,  and  a  score  for  each  item  are  presented  in  later 
sections  of  this  report. 

1 .5  Summary  of  Recommendations 

PEO  C4I  and  OPNAV  should  press  forward  with  development  and  fielding  of  several 
technologies  that  received  high  marks  for  operational  utility:  FASTC2AP,  Global  Trader, 
PANDA,  Tripwire,  and  CMA. 

OPNAV  should  evaluate  the  expressed  concerns,  deficits,  and  return  on 
investments  for  several  technologies  that  received  low  marks  for  operational  utility  or 
system  concerns:  MDA  Data  Sharing  -  Community  of  Interest  (DS-COI),  Electronic 
Maritime  Interdiction  Operation  (E-MIO)  Wireless,  and  MAGNET.  OPNAV  should  also 
consider  placing  technologies  in  reach-back  facilities  (e.g.,  at  ONI  rather  than  at 
NAVCENT)  if  those  technologies  require  robust  technical  support  and  operator 
competence,  that  is,  technologies  with  have  high  utility  but  low  accuracy,  reliability,  or 
usability.  Recommendations  concerning  the  fielding  and  support  process  are  offered. 

In  addition  to  the  recommendations  above  concerning  System  Supportability  and 
Readiness,  we  offer  recommendations  regarding  System  Performance  (re:  usability, 
redundant  functionality,  baseline  specification  functions,  data  sources,  data  source 
education,  common  data,  and  training)  and  Organization/Guidance  (concerning  process 
interoperability,  and  information  flow  impedance).  (See  section  5.0,  below,  for  detailed 
rec  ommendations . ) 

1.6  Future  Assessment  Reeommendations 

We  recommend  that  observations  be  conducted  to  establish  a  baseline  on  MDA 
capabilities  using  current  systems,  in  order  to  better  estimate  the  impact  of  Spiral- 1 
systems.  We  recommend  that  some  experimentation  and  observation  address  other 
technologies  that  promise  to  improve  MDA  effectiveness.  Such  systems  include  CMMA, 
NEPTUNE,  GALE-Eite,  Palaemon,  PANDA,  and  Sea  Watch. 

OPNAV  and  PEO  C4I  should  develop  exercises  that  train  and  test  MDA 
capabilities  (technologies,  tactics,  techniques,  and  procedures  (TTPs),  organizations. 
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etc.),  with  a  particular  focus  on  handling  realistically  large  numbers  of  cooperative, 
intentionally  uncooperative,  and  inattentive  white  vessels. 

Finally,  we  advise  OPNAV  to  ensure  that  future  assessments  capture  data  that 
support  computational,  “what  if’  modeling  of  the  impact  of  new  technologies,  processes, 
manning,  and  organizational  structures.  Such  measurements  will  enable  the  Navy  reuse 
scant  data  productively  and  predictively,  thus  multiplying  the  return  on  each  assessment 
dollar. 
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2.0  Background 

2.1  Objective 

This  assessment  report  fulfills  the  following  statement  of  work,  issued  to  NFS  by  PEO 
C4I; 

1 .  Refine  a  projeet  plan  that  provides  a  eoneept  of  operations  (CONOP)  and  TTP 
around  the  eore  operational  threads  (e.g.,  standard  work  flows,  or  “business 
praetiees”)  to  be  then  further  used  for  operational  field  experimentation  in  Trident 
Warrior  08  (planned  for  execution  in  June  of  2008). 

2.  Specify  measures  and  metrics  related  to  decision  making  and  the  continued 
evolution  of  MDA  system  elements  that  contribute  within  the  GWOT  (Global 
War  On  Terror),  and  are  also  consistent  with  DoD,  JCIDS  experimentation,  and 
acquisition  program  needs. 

3.  Integrate  efforts  across  MDA  working  groups,  brought  together  in  operational 
testing  venues,  under  a  consistent  experiment  design  process  that  will  also  include 
standard  metrics  developed  for  MDA  analysis  of  capabilities  to  ultimately  assess 
Spiral- 1  capabilities  against  fleet  requirements. 

Accordingly,  this  report  (1)  defines  the  current  MDA  workflow  that  new  systems  must 
support  and  how  these  new  technologies  will  impact  the  current  MDA  workflow  (see 
sections  8.4  and  8.5),  (2)  defines  a  measurement  and  assessment  framework  for 
evaluating  MDA  systems  (technological,  procedural,  and  organizational),  and  (3)  applies 
that  framework  in  an  assessment  of  MDA  systems. 

2.2  Motivation 

Maritime  Domain  Awareness  (MDA)  is  a  National  Security  concept  that  relies  on  the 
aggregate  capabilities  of  multiple  government  agencies  such  as  the  Department  of 
Defense  (DoD)  and  Department  of  Homeland  Security  (DHS),  as  well  as  other  federal, 
state,  and  local  agencies  in  order  to  achieve  comprehensive  situational  awareness  of  any 
threat  associated  within  the  Maritime  Domain. 

The  National  Plan  to  Achieve  Maritime  Domain  Awareness  (October  2005) 
defined  the  Maritime  Domain  as  “all  areas  and  things  on,  under,  relating  to,  adjacent  to, 
or  bordering  on  a  sea,  ocean,  or  other  navigable  waterway,  including  all  maritime-related 
activities,  infrastructure,  people,  cargo,  and  vessels  and  other  conveyances.”  Furthermore, 
the  National  Plan  identifies  Nation-state,  terrorist,  transnational  criminal  and  piracy,  and 
environmental  and  social  threats  within  the  Maritime  Domain.  In  order  to  address  these 
threats,  the  National  Plan  requires  the  capabilities  to: 

o  Persistently  monitor  in  the  global  maritime  domain  vessels  and  craft, 
cargo,  vessel  crews  and  passengers,  in  all  identified  maritime  situation 
awareness  areas  of  interest 
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o  Access  and  maintain  data  on  vessels,  facilities,  and  infrastructure 
o  Collect,  fuse,  analyze,  and  disseminate  information  to  deeision  makers  to 
faeilitate  effeetive  understanding 

o  Aeeess,  develop  and  maintain  data  on  MDA-related  mission  performanee. 

The  Department  of  Defense,  following  guidanee  set  forth  from  the  National 
Coneept  of  Operations  for  MDA,  developed  the  Fleet  Coneept  of  Operations  for 
Maritime  Domain  Awareness  (13  Mareh  2007)  and  the  Navy  MDA  Coneept  (29  May 
2007),  whieh  deseribe  the  Fleet  role  in  MDA  and  how  Fleet  eommanders  will  develop 
and  maintain  MDA  to  aeeomplish  Navy  missions  aeross  the  full  Range  of  Military 
Operations  (ROMO). 

Operations  for  MDA  (August  2007)  provide  a  foundation  for  developing 
interageney  and  agency-specifie  polieies,  proeesses,  proeedures,  and  organizational 
relationships  to  align  aetivities  that  eontribute  to  aehieving  MDA  throughout  the  Global 
Maritime  Community  of  Interest  (GMCOI). 

In  a  memorandum  dated  17  May  2007,  the  Seeretary  of  the  Navy  directed  the 
fielding  of  a  prototype  MDA  capability  by  August  2008,  and  established  a  Cross 
Funetional  Team  (CFT)  to  oversee  the  effort.  The  memorandum  direets  the  following  end 
state: 

1 .  Begin  fielding  an  enduring  operational  MDA  eapability. 

2.  Spiral- 1  will: 

a.  Provide  a  eapability  to  the  U.S.  Central  Command  (CENTCOM)  and 
U.S.  Paeifie  Command  (PACOM)  Areas  of  Responsibility  (AORs), 
interagency  partners,  and  seleet  friendly  and  allied  nations. 

b.  Create  a  network  that,  at  multiple  levels  of  security  and  across  multiple 
domains,  will  feed  many  data  streams  into  a  eommon  operational  picture 
(COP)  aeeessible  throughout  the  United  States  Government  and  foreign  or 
Coalition  partners. 

e.  Be  able  to  handle  time  sensitive  maritime  threats, 
d.  Be  designed  for  expansion. 

3.  The  effort  will  be  used  to  resolve  or  develop  new  policy  and  procedures  for 

MDA. 

4.  Subsequent  spirals  will  extend  this  eapability  and  add  functionality. 

The  Deputy  Chief  of  Naval  Operations  (Communieation  Networks)  (N6)  and 

Aeting 

Deputy  Under  Seeretary  of  the  Navy  (DUSN)  were  designated  as  eo-ehairs  of  the  MDA 
CFT.  The  Assistant  Seeretary  of  the  Navy  (Researeh,  Development,  &  Aequisition) 

(ASN  (RDA))  designated  the  Spaee  and  Naval  Warfare  Center’s  (SPAWAR)  Program 
Exeeutive  Offiee  for  Command,  Control,  Communieations,  Computers  and  Intelligenee 
(PEO  C4I)  as  the  Aequisition  Eead  for  delivery  of  the  SECNAV’s  MDA  Prototype. 

Deputy  Under  Seeretary  of  the  Navy  (DUSN)  appointed  the  Naval  Postgraduate 
Sehool  (NPS)  to  assess  the  overall  eapabilities  and  needs  of  MDA,  per  the  seope  of  work 
speeified  above. 

Within  the  NPS  assessment  team,  our  motivation  is  to  assess  and  improve  the 
effeetiveness  of  warfighters  on  MDA  missions. 
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2.3  MDA  Spiral- 1  Technologies 

This  assessment  focuses  on  the  eight  Spiral- 1  technologies,  which  were  tested  in  FY08. 
These  technologies  are: 

•  Comprehensive  Maritime  Awareness  (CMA)  -  The  Naval  Research  Laboratory’s 
enhanced  vessel  tracking  project 

•  Law  Enforcement  Information  Exchange  (EInX)  -  The  Naval  Criminal 
Investigative  Service’s  non-classified  system  for  information  sharing  of  law 
enforcement  information 

•  Global  Trader-  The  Office  of  Naval  Intelligence’s  cargo  data  and  anomaly 
detection  too 

•  Maritime  Global  Network  (MAGNet)  -  The  Coast  Guard’s  intelligence  program 
backbone  database  with  enhanced  anomaly  detection  for  people 

•  Tactical  E-MIO  System  Wireless  -  An  E-MIO  data  collection  and  transfer  system 

•  EAST2CAP  -  A  common  maritime  operating  picture  that  allows  the  watchstander 
to  construct,  implement  and  reconfigure  search  agents 

•  Tripwire  -  ONI’s  threat  detection  tools 

•  Google  Earth  -  A  commercial  toolset  for  fusing  data  and  displaying  it  on  a  globe 

•  MDA  Data  Sharing  Community  of  Interest  (DS  COI)  -  A  system  for 
disseminating  AIS  data 

Data  concerning  other  non-Spiral-I  MDA  systems  (e.g.,  PANDA)  were  also  collected 
during  the  test  events,  and  these  results  are  also  included  in  this  report.  . 

2.4  Summary  of  Next  Sections 

In  the  following  sections  we  describe  our  method  of  analysis  (Section  3.0), 
summarize  our  findings  (Section  4.0),  present  recommendations  (Section  5.0),  and 
present  detailed  findings  in  each  assessment  area  (Section  6.0).  In  sections  4.0  and  6.0, 
we  present  the  average  scores  in  each  category  as  assessments  of  the  MDA  technology, 
as  we  interpret  these  from  the  categorized,  qualitative  data.  Einally,  appendices  present 
raw  data  and  other  material  of  interest. 
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3.0  Assessment  Method 


The  assessment  method  used  for  this  MDA  effort  extends  the  teehniques  developed  and 
refined  in  the  Trident  Warrior  series  of  exereises  sinee  2002.  The  elements  of  this 
method,  in  brief,  were; 

1 .  Define  a  framework  for  assessment  in  the  domain:  MDA 

2.  Colleet  data  from  test  and  experimental  venues 

3.  Categorize  data  using  the  framework,  assign  seores  to  eategorized  data,  and 
summarize  the  data 

We  describe  these  steps  in  more  detail  below. 

3 . 1  Assessment  Framework 

The  MDA  assessment  framework  specifies  a  three-level  hierarchy  of  assessment  areas, 
metrics  to  evaluate  performance,  and  specific  measures.  This  framework  is  designed  to 
address  technological  systems,  organizational  structures  and  processes,  policy  or 
guidance  and  other  factors  that  ensure  a  robust  mission  capability. 

The  assessment  framework  addresses  five  distinct  assessment  areas,  and  two 
levels  of  sub-areas  within  them.  (For  a  complete  structure  description  see  section  14.0.) 
The  five  assessment  areas  are: 

o  System  Performance  concerns  how  well  a  system  performs  its  functions,  its 
support  of  MDA  operations,  warfighter  acceptance,  automation,  and  system 
management  and  security  functions. 

o  Operations  Performance  addresses  the  quality  of  knowledge  management 
concerning  Vessels  of  Interest  (Vols),  MDA  intelligence,  surveillance,  and 
reconnaissance  (ISR),  and  MIO. 

o  Warfighter  Performance  focuses  on  operator  acceptance  and  understanding  of  the 
MDA  mission,  as  well  as  unit  and  individual  capability  to  execute  that  mission. 

o  Organization/Guidance  focuses  on  the  fit  of  organizational  structures  and 
processes  to  the  MDA  mission  (including  MHQ  w/  MOC),  the  sufficiency  of 
agreements  between  entities,  and  the  adequacy  of  guidance  within  entities. 

o  Supportability  and  Readiness  concerns  factors  that  ensure  MDA  Spiral- 1  systems 
are  robust  and  reliable.  (This  assessment  area  is  primarily  reported  by  PEO  C4I.) 

The  full  hierarchy  of  assessment  areas  is  represented  next  in  Table  1. 
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Table  1.  MDA  Assessment  Areas  and  Structure. 


System  Performance 

Operations  Performance 

Technical  Performance 

Knowledge  Processes 

Information  Retrieval 

Information  Retrieval 

Information  Processing 

Vol  Development 

Information  Sharing 

Vol  Tracking 

Operator  Configurable 

Information  Sharing 

Interoperability 

ISR 

Operations  Support 

Planning 

System  Utility 

Execution 

Standards  and  Guidelines 

PED 

Warfighter  Acceptance 

MIO 

System  Utility 

Planning 

Human-System  Interaction 

Execution 

System  Usage 

Assessment 

System  Training 

Automation 

Organization/Guidance 

Alerts 

MDA  Compatibility 

Information  Processing 

Organization  Alignment 

Smart  Pull 

Process  Alignment 

System  Management  and  Security 

MHQ/MOC  Compatibility 

Organization  Alignment 

Warfighter  Performance 

Process  Alignment 

MDA  Mission 

Agreements 

Mission  Understanding 

Information  Sharing 

Mission  Acceptance 

Shared  Operations 

Unit  Performance 

Guidance 

Manning 

CONOPS 

Activities 

TTP/SOP 

Training 

Standing  Orders 

Human  Performance 

Tasks 

System  Supportability  and  Readiness 

Training 

PEO  Provided 

Capability  indicators  (or  assessment  metries)  have  been  defined  for  each  of  the 
Assessment  Areas.  These  indicators  include  both  attributes  and  their  measures  taken 
from  those  used  for  the  Naval  Network  Warfare  Command  (NAVNETWARCOM) 
Capabilities  Based  Analyses  for  NCO,  eommand  and  control  (C2),  and  BA.  Section  14.0 
presents  this  full  attribute  strueture. 

Table  2  shows  the  attributes  used  for  MDA  assessment;  not  all  are  used  for  each 
Assessment  Area.  The  prineipal  indicators  are  Effeetiveness  and  Military  Utility. 
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Table  2,  MDA  Capability  Indicators. 


Area 

Effective 

Utility 

Area 

Effective 

Systems 

Accessible 

Capable 

Reliable 

Usable 

Improved 

Needed 

Applicable 

Wanted 

Warfighter 

Capable 

Reliable 

Operations 

Accessible 

Capable 

Reliable 

Organization 

/Guidance 

Accessible 

Capable 

Usable 

Specific  measures  and  data  are  defined  for  each  of  the  capability  indicators.  Examples 
are  shown  in  Table  3  and  the  full  list  is  in  Section  14.0. 


Table  3,  MDA  Assessment  Metrics  Example, 


MOE 

MOP  or  MOU 

SpiraM  System  Performance  (each  system) 

Technical 

Performance 

Improved:  -5  to  +5  rating  of  improvement  over  existing  systems,  by  system 
aspect. 

Needed:  system  fiiis  a  gap  in  existing  capabiiities,  Y/N. 

Applicable:  system  is  appiicabie  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  avaiiable. 

Information 

Retrieval 

Accessible:  roll-up  of  information  accessibility. 

Available:  %  of  time  information  is  available. 

Efficient:  number  of  steps  to  access  information. 

Capable:  roll-up  of  capability  to  retrieve  required  information. 

Sufficient:  %  of  information  needed  for  assessment. 

Timely:  time  required  to  retrieve  information. 

Metric  Attributes  are  shown  in  bold  and  their  measures  in  plain  text.  Use  of  the  structure 
presented  in  Table  3  provides  a  consistent  approach  to  MDA  assessment  and  reporting 
across  test  venues.  This  structure  will  be  used  to  correlate  and  fuse  results  from  a  variety 
of  sources  throughout  the  life  of  the  assessment  program. 
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3.2  Data  Collection 


The  data  analyzed  in  this  assessment  were  colleeted  during  multiple  site  visits  and  several 
Spiral  -1  system  test  events: 

1 .  Workflow  analysis  and  Proeess  Engineering  Workshop  (PEW) 

a.  An  overview  of  MDA  aetivities  was  eondueted  during  site  visits  to 
NAVCENT,  ONI,  Seeond  Eleet,  Third  Eleet,  Eifth  Eleet,  and  Sixth  Eleet. 
The  workflow  for  MDA  as  eurrently  executed  was  documented  in  these 
visits  (see  Appendix  1). 

b.  The  workflow  was  validated  in  the  PEW  held  at  the  Naval  Postgraduate 
School,  Monterey,  CA,  15-17  Jan  2008. 

c.  The  workflow  was  mapped  to  the  MHQ  w/MOC  core  processes  in  a 
workshop  in  Norfolk,  VA,  in  29  January  2008. 

2.  TRRLOE  (Technical  Risk  Reduction  Eimited  Objective  Experiment) 

a.  Held  at  SPAWAR,  Eab  140,  San  Diego,  CA,  2-6  June  2008. 

b.  hirst  simultaneous  testing  of  multiple  MDA  technologies  using  Eleet 
participants 

3.  VBSS  (Visit  Board  Search  and  Seizure)  School: 

a.  Held  in  San  Diego,  CA,  19  June  2008 

b.  Observed  EMIO  usage  and  effectiveness 

4.  Trident  Warrior  08 

a.  Held  aboard  multiple  Navy  ships  to  test  a  myriad  of  newly  developed 
technologies  and  policies,  15  Jun  to  15  July  2008 

5.  Empire  Challenge  08 

a.  Held  at  China  Eake  Naval  Station  and  in  San  Erancisco  Harbor  as  well  as 
multiple  other  multiple  sites,  7-31  July  2008 

6.  EAIRGAME: 

a.  Held  at  simultaneous,  multiple  sites  (NMIC,  NAVCENT,  PACELT, 
MIECPAC,  MIECEANT,  SSC-SD,  andNCIS  MTAC),  15-18  July  2008 

b.  Used  as  the  primary  source  for  COMOPTEVEOR  to  perform  the  Quick 
Reaction  Assessment  (QRA)  on  all  MDA  technologies 

The  data  gathered  in  these  venues  were  largely  qualitative  -  including  survey  comments, 
chat  logs,  data  collectors’  observations,  and  interview  notes  -  though  some  quantitative 
survey  data  were  also  collected.  As  noted  above,  the  data  largely  concerned  the  Spiral- 1 
systems  evaluated  in  most  of  these  venues.  However,  data  were  also  collected  on  other 
systems  with  potential  MDA  utility.  In  sum,  the  systems  assessed  were:  CMA,  E-MIO 
Wireless,  EASTC2AP,  Global  Trader,  Google  Earth,  EiNX,  MAGNET,  MASTER,  MDA 
DS  COI,  MIDAS,  PANDA,  and  Tripwire. 

3.3  Analysis  Method 

The  data  were  aggregated  across  all  venues.  Each  datum  was  assigned  to  one  or  more 
areas  (e.g..  System  Performance)  of  the  assessment  framework  (depicted  in  Table  1)  at 
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the  lowest  level  of  detail  (e.g.,  Technieal  Performance,  or  sub-sub-areas).  Each  item  in 
each  category  was  rated  to  indicate  that  it  described  a  strength  (score  =  3),  a  concern 
(score  =  2),  or  a  deficiency  (score  =  1)  in  an  MDA  system,  suite  of  systems,  or 
deployment  concept  for  systems.  Items  scored  as  concerns  (2)  were  relatively  minor  or 
could  be  addressed  through  revisions  to  training  or  interface  design.  Items  scored  as 
deficiencies  (1)  concerned  missing  or  inoperable  functions  critical  to  the  MDA  mission. 
Average  scores  were  computed  for  each  assessment  area.  These  scores  focused  our 
interpretation  of  the  qualitative  data.  We  have  given  special  emphasis  to  describing 
reported  deficiencies. 

It  is  important  to  note  that  many  of  the  reports  are  from  a  single  source  and,  thus,  may 
reflect  the  personal  biases  of  those  sources.  Note  also  that  the  quantitative  findings 
reported  here  were  developed  by  the  NPS  research  team  from  qualitative  reports,  and  thus 
reflect  our  interpretation  rather  than  the  interpretation  of  diverse  operational  experts  or 
technology  experts. 

In  the  next  sections,  we  summarize  our  findings  from  each  of  the  assessment  areas. 
Following  this, ,  we  present  recommendations  based  on  these  findings.,  and  then  describe 
the  findings  in  detail. 
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4.0  MDA  Capabilities  Results  Summary 

4.1  Introduction 

This  section  presents  a  qualitative  and  quantitative  summary  of  MDA  capabilities  within 
the  assessment  areas  included  in  the  assessment  framework  developed  for  this  effort.  It 
summarizes  findings  across  the  assessment  venues.  The  data  that  support  these 
summaries  are  presented  in  subsequent  chapters. 

The  data  presented  here  comprise  a  compilation  of  194  observations,  warfighter 
comments,  and  survey  results.  These  were  each  assigned  to  one  or  more  of  the  relevant 
assessment  sub-areas.  Many  items  were  placed  in  multiple  categories,  resulting  in  a  total 
of  304  assessments.  Items  were  scored  on  the  three-point  scale  described  above.  Average 
scores  drove  interpretation  of  the  qualitative  data. 

4.2  Results  by  Assessment  Area 

The  data  generated  across  the  MDA  assessment  venues  focused  on  the  use  of  MDA 
Spiral- 1  technologies  for  developing  and  tracking  VOIs  and  conducting  MIO  operations. 
Accordingly,  most  of  the  data  were  categorized  in  the  system  performance  assessment 
area  (219  items)  and  the  operations  performance  assessment  area  (70  items).  In  both 
areas,  the  average  assessment  score  was  relatively  high;  2.4  out  of  3.  No  items  concerned 
warfighter  performance  independent  of  the  MDA  technologies,  and  no  assessment  is 
made  in  this  area.  Only  eight  items  concerned  the  organization/guidance  assessment  area, 
and  seven  items  concerned  the  system  supportability  and  readiness  area.  Assessment 
scores  were  lowest  in  these  low-frequency  categories,  largely  due  to  overall  concerns 
about  the  utility  and  supportability  of  new  technologies  from  organizations  being  tasked 
with  the  new  MDA  mission.  The  frequency  of  reference  to  Spiral- 1  and  other  MDA 
technologies  is  presented  in  Table  4. 

Table  4,  Distribution  of  data  by  assessment  area. 


Area 

Items 

Avg  Score 

Operations  Performance 

70 

2.4 

Organization/  Guidance 

8 

2.1 

System  Performance 

219 

2.4 

System  Supportability  and  Readiness 

7 

2.0 

Total  /  Average 

304 

2.4 

The  data,  which  include  warfighter  comments,  observations,  interview  notes,  and  survey 
results,  specified  123  strengths  of  the  tested  MDA  systems,  166  concerns,  and  15 
deficiencies  (see  Table  5).  Ten  deficiencies  concerned  System  Performance,  and  five 
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concerned  Operations  Performanee.  Because  deficiencies  are  the  most  critieal  data,  we 
address  these  in  depth  in  the  relevant  assessment  areas,  below. 

Table  5.  Distribution  of  data  by  score  by  assessment  area. 


Count  of  Score 

Score 

Assessment  Area 

1:  Deficiency 

2:  Concern 

3:  Strength 

Total 

Operations  Performance 

5 

33 

32 

70 

Organization/  Guidance 

7 

1 

8 

System  Performance 

10 

119 

90 

219 

System  Supportability  and 
Readiness 

7 

7 

Total 

15 

166 

123 

304 

We  turn  now  to  a  qualitative  assessment  of  eaeh  assessment  area  for  whieh  there  was 
data. 

4.2.1  System  Performance 

Of  219  items  that  concerned  System  Performance,  10  specified  potential  deficieneies  of 
MDA  system  performance.  Three  of  these  eoncerned  the  inability  to  specify  baselines 
against  which  to  compare  observed  vessel  behavior.  Two  items  eoneemed  perceived 
inadequacy  of  data  quality  and  availability  and  two  items  eoneemed  laek  of  eonnectivity 
of  MIO  teehnology.  The  remaining  three  items  concerned  specifie  features,  data  sharing, 
or  security.  Because  deficieneies  are  particularly  important  in  assessment,  we  present  the 
detailed  eomments  here. 


14 


Table  6,  Potential  deficiencies  regarding  System  Performance. 


Sub-Sub-Area 

Technology 

Item 

System  Performance: 
System  Management 
and  Security:  System 
Management  and 
Security 

CMA 

CMA:  Users  sometimes  lost  track  of  the  original  security 
classification  of  the  information  they  wished  to  disseminate.  This 
increased  the  likelihood  of  a  security  violation  as  a  result  of 
passing  classified  information  on  the  wrong  domain. 

System  Performance: 
Technical 

Performance: 
Information  Sharing 

Google  Earth 

Google  Earth  has  no  embedded  collaboration  tool  included,  thus 
it  was  not  possible  to  send  information  from  Google  Earth  to 
other  systems. 

System  Performance: 
Automation:  Alerts 

All  Spiral- 1 

Spiral- 1  did  not  automatically  establish  or  display  threat 
assignments  based  upon  a  user-defined  alert. 

System  Performance: 
Automation:  Alerts 

CMA& 

FASTC2AP 

Spiral- 1  tools  did  not  provide  additional  capability  to  establish 
baseline  normal  civil  maritime  operations  worldwide  and  threat 
assessment  criteria.  CMA  and  FASTC2AP  could  alert  based 
upon  a  geographic  point/area/proximity,  but  did  not  support  alerts 
employing  algorithms  based  upon  baseline  maritime  operations. 

System  Performance: 
Operations  Support: 
System  Utility 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  Although  wireless,  the  TED 
devices  were  required  to  be  in  the  vicinity  of  the  TEMP  in  order 
to  download  data  captured  during  the  boarding.  The  radio 
frequency  (RF)  signals  were  not  storing  enough  to  transmit  data 
when  team  members  were  below  decks.  (VBSS  School) 

System  Performance: 
Operations  Support: 
System  Utility 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  While  mobility  of  the  Tactical 
EMIO  Device  (TED)  was  a  clear  advantage,  enabling  the 
collection  of  data  from  multiple  locations  within  the  vessel,  one 
limitation  noted  was  that  the  TED  must  be  within  the  vicinity  of 
the  Tactical  EMIO  Maritime  PC  (TEMP)  to  download  the  data 
captured.  Although  wireless,  the  TED  devices  were  required  to 
be  in  the  vicinity  of  the  TEMP  in  order  to  download  data  captured 
during  the  boarding.  The  radio  frequency  (RF)  signals  were  not 
strong  enough  to  transmit  data  when  team  members  were  below 
decks.  The  Maritime  BGAN  EMIO  Terminal  (MBET)  device,  in 
turn,  failed  to  transfer  data  due  to  environmental  issues  and  weak 
RF  Signal  range  of  the  commercial  satellite.  Contractors 
eventually  departed  the  target  vessel  and  drove  inland  with  the 
TEMP  and  MBET  device  to  acquire  a  stronger  signal.  The  MBET 
link  was  then  acquired  and  successfully  transmitted  data  from  the 
TEMP  device.  Contractors  asserted  that  the  satellite  connectivity 
will  not  be  a  concern  in  the  current  AOR.  Also,  the  boarding 
officer  was  not  able  to  demonstrate  the  transfer  of  data  via  the 
Maritime  BGAN  EMIO  Terminal  (MBET)  due  to  the  satellite 
connectivity.  (VBSS  School) 

System  Performance: 
Automation:  Alerts 

All  Spiral- 1 

The  ability  to  capture  and  store  baseline/normal  maritime 
movement  patterns  was  not  observed.  Spiral- 1  tools  did  not  alert 
users  to  deviations  from  normal  route  or  behavior  patterns. 

System  Performance: 
Technical 

Performance: 
Information  Retrieval 

CMA 

The  user’s  ability  to  monitor  vessel,  person,  and  cargo  data  was 
severely  degraded  by  gaps  in  track  data  coverage.  When  a  node’s 
CMA  server  was  down,  or  data  was  not  transmitted,  the  data  not 
received  was  not  recoverable.  The  Naval  Research  Laboratory 
(NRL)  limited  the  National  Technical  Means  (NTM)  data  source 
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input  to  CMA  to  14  hours  per  day  and  fdtered  the  data  that  was 
provided.  This  resulted  in  a  gap  of  data  which  had  a  negative 
impact  across  all  AORs. 

System  Performance: 
Technical 

Performance: 
Information  Sharing 

All  Spiral- 1 

There  were  significant  differences  in  information  available  at 
different  nodes. 

Some  119  items  expressed  reparable  eoncerns  with  MDA  systems.  By  far  the  largest 
group  of  these  (33  items)  addressed  the  completeness,  correctness,  conflicts,  and 
timeliness  of  data  or  data  processing.  Usability  of  maps,  search,  alerts,  and  other  features 
was  cited  in  24  items.  Some  20  items  addressed  problems  with  training  availability,  fit  to 
local  needs,  or  speed.  Operational  utility  was  a  concern  in  7  cases,  and  the  redundancy  of 
technologies  was  an  issue  in  5. 

Of  the  90  strengths  cited  in  operator  comments,  observations,  and  survey  results  (see 

Table  5), 

27  concerned  operational  utility  for  VOI  detection  and  tracking  and  for  MIO  operations. 
Usability  of  search,  maps,  and  other  features  accounted  for  18  items.  The  speed  of 
training  (6  items)  was  also  an  indirect  validation  of  the  usability  of  the  technologies.  Nine 
items  concerned  the  value  of  data  fusion  capabilities  of  these  technologies.  The  speed  of 
the  technologies  was  cited  in  6  items. 

4.2.2  Operations  Performance 

Of  70  items  that  concerned  Operations  Performance,  five  were  potential  deficiencies.  All 
but  one  item  concerned  E-MIO  connectivity  (reception  or  transmission)  problems.  The 
remaining  item  concerned  gaps  in  track  coverage  by  CMA  due  to  failures  of  CMA 
servers.  (Note  that  one  of  the  five  items  in  Table  7  was  coded  in  two  assessment 
categories). 
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Table  7.  Potential  deficiencies  regarding  Operations  Performance, 


Sub-Sub-Area 

Tecbnology 

Item 

Operations  Performance: 
Knowledge  Processes: 

Vol  Development  & 
Operations  Performance: 
MIO:  Execution 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  While  mobility  of  the  Tactical 
EMIO  Device  (TED)  was  a  clear  advantage,  enabling  the 
collection  of  data  from  multiple  locations  within  the  vessel, 
one  limitation  noted  was  that  the  TED  must  be  within  the 
vicinity  of  the  Tactical  EMIO  Maritime  PC  (TEMP)  to 
download  the  data  captured.  Although  wireless,  the  TED 
devices  were  required  to  be  in  the  vicinity  of  the  TEMP  in 
order  to  download  data  captured  during  the  boarding.  The 
radio  frequency  (RF)  signals  were  not  strong  enough  to 
transmit  data  when  team  members  were  below  decks.  The 
Maritime  BGAN  EMIO  Terminal  (MEET)  device,  in  turn, 
failed  to  transfer  data  due  to  environmental  issues  and  weak 

RF  Signal  range  of  the  commercial  satellite.  Contractors 
eventually  departed  the  target  vessel  and  drove  inland  with  the 
TEMP  and  MEET  device  to  acquire  a  stronger  signal.  The 
MEET  link  was  then  acquired  and  successfully  transmitted 
data  from  the  TEMP  device.  Contractors  asserted  that  the 
satellite  connectivity  will  not  be  a  concern  in  the  current  AOR. 
Also,  the  boarding  officer  was  not  able  to  demonstrate  the 
transfer  of  data  via  the  Maritime  BGAN  EMIO  Terminal 
(MEET)  due  to  the  satellite  connectivity.  (VBSS  School) 

Operations  Performance: 
Knowledge  Processes: 

Vol  Tracking 

CMA 

The  user’s  ability  to  monitor  vessel,  person,  and  cargo  data 
was  severely  degraded  by  gaps  in  track  data  coverage.  When  a 
node’s  CMA  server  was  down,  or  data  was  not  transmitted,  the 
data  not  received  was  not  recoverable.  The  Naval  Research 
Laboratory  (NRL)  limited  the  National  Technical  Means 
(NTM)  data  source  input  to  CMA  to  14  hours  per  day  and 
filtered  the  data  that  was  provided.  This  resulted  in  a  gap  of 
data  which  had  a  negative  impact  across  all  AORs. 

Operations  Performance: 
MIO:  Execution 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  Although  wireless,  the  TED 
devices  were  required  to  be  in  the  vicinity  of  the  TEMP  in 
order  to  download  data  captured  during  the  boarding.  The 
radio  frequency  (RF)  signals  were  not  storing  enough  to 
transmit  data  when  team  members  were  below  decks.  (VBSS 
School) 

Operations  Performance: 
MIO:  Execution 

E-MIO 

There  were  no  positions  provided  with  the  latent  print 
messages  and  for  the  cave  collection,  there  was  no  Seek  ID 
Global  Positioning  System  (GPS)  reception  and  therefore  no 
position  was  included  in  the  biometric  messages. 

Of  33  items  coded  as  concerns  (i.e.,  a  score  of  2),  the  most  frequent  themes  were  the 
completeness,  correctness,  conflicts,  timeliness  of  data  and  data  processing  (12  items), 
usability  (10  items),  and  training  (5  items). 

Of  32  items  coded  as  strengths,  13  concerned  operational  utility  for  VOI;  10  concerned 
usability;  4  focused  on  the  utility  of  alerts;  and  the  rest  addressed  miscellaneous  issues. 
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4.2.3  Warfighter  Performance 


No  items  concerned  warfighter  performance  independent  of  the  MDA  technologies,  and 
thus  no  assessment  is  made  in  this  area. 


4.2.4  Organization/Guidance 

Very  few  items  (8  in  total)  concerned  the  assessment  area  of  Organization  and  Guidance. 
Of  these,  none  rose  to  the  level  of  a  potential  deficiency. 

Seven  items  were  coded  as  concerns.  Two  each  concerned  the  constraints  imposed  by 
policies  and  agreements;  coordination  of  MDA  execution  and  ONI  intelligence  processes; 
and  training.  One  addressed  the  challenge  of  customizing  solutions  to  fit  the  local 
missions  of  organizations. 

One  item  was  scored  as  a  strength  (i.e.,  a  score  of  3).  It  concerned  the  alignment  of  MDA 
tasks  with  MHQ  w/  MOC  processes  in  the  Process  Alignment  Workshop  described  in 
section  Appendix  B:  Process  Engineering  &  Alignment  Workshop  Results  (section  8.0). 

4.2.5  System  Supportability  and  Readiness 

Seven  items  addressed  System  Supportability  and  Readiness.  All  of  these  were  areas  of 
concern,  either  regarding  the  competency  or  size  of  staff  (4  items),  or  potential 
deficiencies  in  infrastructure  (2  items)  or  variance  between  facilities  (1  item)  that  might 
hinder  fielding  and  use  of  MDA  solutions. 

4.3  Results  by  Assessment  Sub- Area 

In  this  section,  we  summarize  findings  in  each  assessment  sub-area.  A  summary  of  the 
quantitative  assessment  is  presented  in  Table  8,  followed  by  a  qualitative  assessment  that 
summarizes  the  strengths,  concerns,  and  deficiencies  observed  in  the  various  MDA 
assessment  venues. 

Note,  in  the  quantitative  assessment,  that  areas  of  greatest  concern  (lowest  average 
scores)  were  Operations  Performance  for  MIO,  several  areas  of  Organization/Guidance 
(Guidance,  MDA  Compatibility,  and  System  Management  and  Security),  and  System 
Supportability  and  Readiness.  The  most  highly  rated  sub-areas  were  Operations 
Performance:  Knowledge  Processes,  Organization/Guidance:  MHQ/MOC  Compatibility, 
and  System  Performance:  Operations  Support.  Finer-grained  categorization,  a  synopsis, 
and  a  score  for  each  item  are  presented  in  later  sections  of  this  report. 
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Table  8,  Average  assessment  scores  by  sub-area. 


Assessment  Sub-Area 

Average  score 

Operations  Performance:  Knowledge  Processes 

2.5 

Operations  Performance:  MIO 

1.8 

Organization/Guidance:  Agreements 

2.0 

Organization/Guidance:  Guidance 

2.0 

Organization/Guidance:  MDA  Compatibility 

2.0 

Organization/Guidance:  MHQ/MOC  Compatibility 

2.5 

System  Performance:  Automation 

2.4 

System  Performance:  Operations  Support 

2.5 

System  Performance:  System  Management  and  Security 

2.0 

System  Performance:  Technical  Performance 

2.3 

System  Performance:  Warfighter  Acceptance 

2.4 

System  Supportability  and  Readiness 

2.0 

Average 

2.4 

4.3.1  System  Performance:  Technical  Performance 


Sub-Sub-Area 

Total 

System  Performance:  Technical  Performance:  Information  Processing 

1 

System  Performance:  Technical  Performance:  Information  Retrieval 

54 

System  Performance:  Technical  Performance:  Information  Sharing 

21 

System  Performance:  Technical  Performance:  Interoperability 

5 

Average 

81 

The  majority  of  the  81  items  eoneerning  System  Performance:  Technical  Performance 
involved  Information  Retrieval  (54  items),  all  but  30  of  which  addressed  concerns  (i.e., 
score  of  2)  about  the  completeness,  correctness,  conflicts,  timeliness  of  data  and  data 
processing.  Usability  strengths  and  concerns  were  the  topic  of  8  items.  Also  noted,  with  5 
items  each,  were  data  fusion  (generally  a  strength)  and  speed  of  data  processing  (which 
received  mixed  assessments).  The  remaining  items  addressed  various  issues. 

Of  the  remaining  items,  21  concerned  Information  Sharing.  More  specifically,  7 
addressed  the  completeness,  correctness,  conflicts,  timeliness  of  data  and  data  processing. 
Three  each  concerned  data  sharing  and  transmission. 

Five  items  addressed  Interoperability  of  systems.  Of  these  items,  three  addressed 
requirements  for  specific  software  (e.g.,  Java,  Direct-X),  and  two  concerned  the 
availability  of  data  within  or  between  nodes. 
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One  item  addressed  Information  Proeessing,  speeifioally  the  need  for  automated 
statistieal  analysis  of  data  gaps  to  identity  potential  new  sourees  of  information  and  drive 
new  eolleetions.  This  eapability  -  data-driven  eolleetion  planning  -  is  being  developed  in 
research  efforts  for  ground  warfare,  and  could  be  extended  to  MDA. 


4.3.2  System  Performance:  Operations  Support 

Of  21  items  that  concerned  System  Performance:  Operations  Support,  20  concerned  the 
utility  of  specific  technologies  to  operations.  As  noted  above,  E-MIO  technology 
exhibited  a  potential  deficiency  with  respect  to  connectivity.  Most  items  in  this  area  were 
strengths  or  concerns  about  the  general  value  of  these  tools.  One  item  concerned  the  lack 
of  TTPs  and  SOP  (Standards  and  Guidelines)  for  applying  MDA  technologies. 

4.3.3  System  Performance :  W arfighter  Acceptance 

Some  85  items  addressed  Warfighter  Acceptance  of  specific  technologies.  Of  these,  41 
concerned  Human-System  Interaction,  the  majority  focusing  on  usability  of  agents,  alerts, 
briefs,  help,  maps,  menus,  and  search.  Five  of  these  concerned  CMA’s  data  fusion 
capabilities,  which  were  viewed  as  a  strength  in  most  cases.  Training  was  addressed  in  22 
items,  which  was  seen  as  fast  but  often  incomplete.  System  Utility  was  addressed  by  18 
items,  with  an  emphasis  on  acceleration  of  operational  tasks,  and  benefits  to  situational 
awareness.  However,  MDA  DS  COI,  LiNX,  and  Google  Earth  were  each  viewed  once 
with  concern  (i.e.,  a  score  of  2)  for  being  redundant  with  other  solutions.  Strategies  for 
using  the  systems  were  seen  to  be  lacking  in  four  cases  in  an  area  we  call  System  Usage. 

4.3.4  System  Performance:  Automation 

Automation  was  addressed  in  27  items.  Of  these,  26  concerned  Alerts,  specifically  their 
strong  operational  utility  but  mixed  usability.  One  item,  concerning  Information 
Processing,  addressed  MASTER’S  strong  automation  to  capture  data  on  VOIs,  relative  to 
current  solutions. 

4.3.5  System  Performance:  System  Management  and  Security 

Of  five  System  Management  and  Security  issues,  4  were  concerns  about  keeping  systems 
up  and  running,  and  one  -  a  potential  deficiency  -  concerned  loss  of  the  original  security 
classification  of  information  to  be  disseminated. 

4.3.6  Operations  Performance:  Knowledge  Processes 

VOI  Development  was  the  focus  of  52  of  the  62  items  concerning  Operations 
Performance:  Knowledge  Processes.  In  this  area,  usability  (17  items),  operational  utility 
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(14  items),  and  the  completeness,  correctness,  conflicts,  timeliness  of  data  and  data 
processing  were  the  most  frequent  topics. 

VOl  Tracking  accounted  for  the  remaining  10  items,  and  similar  concerns  arose  in  this 
area;  completeness,  correctness,  conflicts,  timeliness  of  data  and  data  processing  (4 
items),  and  mixed  usability  (2  items)  were  the  most  frequently  cited  issues. 

4.3.7  Operations  Performance:  MIO 

In  the  area  of  MIO  Execution,  inadequate  training  was  a  concern  in  3  of  8  cases; 
inadequate  connectivity  was  a  potential  deficiency  in  2. 

4.3.8  Organization/Guidance:  MDA  Compatibility 

The  alignment  of  ONI  with  MDA  activities  was  a  concern  in  2  items. 

4.3.9  Organization/Guidance:  MHQ/MOC  Compatibility 

Two  items  concerned  compatibility  of  MDA  and  MHQ  with  MOC.  Process  Alignment 
between  MDA  and  MHQ  with  MOC  was  seen  as  a  strength  in  one  item,  due  to  the 
Process  Alignment  Workshop.  However,  local  Organization  Alignment  with  MDA,  that 
is,  the  flexibility  of  MDA  processes  to  local  needs,  was  a  concern  in  one  case. 

4.3.10  Organization/Guidance:  Agreements 

Two  items  concerned  the  assessment  area  Organization/Guidance:  Agreements,  and  both 
were  concerns  (i.e.,  rating  of  2)  about  restrictions  imposed  by  data  sharing  agreements 
about  classified  information  or  data  concerning  U.S.  citizens. 

4.3.11  Organization/Guidance:  Guidance 

Two  items  specified  concerns  (i.e.,  rating  of  2)  about  the  need  to  develop  better  MDA 
CONOPS  and  TTP/SOP. 

4.3. 12  System  Supportability  and  Readiness 

Seven  items  concerned  System  Supportability  and  Readiness.  Of  these,  4  were  concerns 
(i.e.,  rating  of  2)  about  the  number  or  competency  of  staff,  and  3  identified  concerns 
about  the  adequacy  of  infrastructure  to  support  MDA  technologies  generally. 

4.4  Results  by  Technology 
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In  this  section,  we  summarize  the  scores  for  each  MDA  technology  cited  in  the  findings  - 
including  Spiral- 1  technologies  and  others  -  at  the  lowest  assessment  level.  For  further 
detail  concerning  each  technology,  we  refer  the  reader  to  the  reports  sections,  below,  in 
which  we  present  findings  in  each  of  the  assessment  areas. 

The  majority  of  the  data  gathered  across  venues  concerned  a  specific  technology  (a  total 
of  257  items),  usually  CMA  (136  items).  A  minority  of  the  data  concerned  no  specific 
MDA  technology,  or  an  unspecified  suite  of  technologies  (47  items)  (see  Table  9).  Note 
that  the  list  of  technologies  includes  some  systems  that  are  not  Spiral- 1  products,  but  that 
were  evaluated  in  the  various  MDA  assessment  venues. 

Those  technologies  that  scored  lowest,  on  the  average  (see  Table  9),  were  Global  Trader, 
for  perceived  incompleteness  of  data  and  lack  of  data  replication;  MAGNET,  for  lack  of 
data  due  to  data  sharing  agreements  concerning  U.S.  citizens;  MDA  DS  COI  for  lack  of 
usability  or  training;  and  E-MIO  wireless  for  connectivity  problems.  Those  that  scored 
highest  were  Tripwire,  for  its  alerting  and  provision  of  contextual  information,  and 
PANDA,  for  the  operational  utility  of  alerts  and  quality  of  explanations  of  deviations  that 
it  presented. 


Table  9,  Distribution  of  data  by  technology. 


Technology 

Frequency 

Average  Score 

All  Spiral- 1 

47 

2.1 

CMA 

136 

2.4 

E-MIO  Wireless 

23 

2.1 

EASTC2AP 

13 

2.4 

Global  Trader 

5 

2.2 

Google  Earth 

10 

2.5 

EiNX 

5 

2.4 

MAGNET 

2 

2.0 

MASTER 

33 

2.3 

MDA  DS  COI 

5 

2.0 

MIDAS 

5 

2.6 

PANDA 

13 

2.9 

Tripwire 

7 

2.9 

Total/ Avg 

304 

2.4 

A  more  detailed  assessment  is  presented  in  Table  10,  which  indicates  the  average  scores 
for  MDA  technologies  at  the  finest  level  of  assessment  detail. 

Table  10.  Average  assessment  scores  for  MDA  technologies. 
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Sub -Sub -Area 

CMA 

E-MIO  Wireless 

FASTC2AP 

Global  Trader 

Google  Earth 

LiNX 

MAGNET 

MASTER 

MDA  DS  COI 

MIDAS 

PANDA 

Tripwire 

All  MDA  Tech 

Average 

Operations  Performance: 
Knowledge  Processes: 

Vol  Development 

2.5 

1.0 

2.0 

3.0 

2.1 

2.0 

3.0 

3.0 

3.0 

2.5 

2.5 

Operations  Performance: 
Knowledge  Processes: 

Vol  Tracking 

2.3 

3.0 

2.4 

Operations  Performance: 
MIO:  Execution 

1.8 

1.8 

Organization/Guidance : 
Agreements: 

Information  Sharing 

2.0 

2.0 

2.0 

Organization/Guidance : 
Guidance:  CONOPS 

2.0 

2.0 

Organization/Guidance : 
Guidance:  TTP/SOP 

2.0 

2.0 

Organization/Guidance : 
MDA  Compatibility: 
Organization  Alignment 

2.0 

2.0 

Organization/Guidance : 
MHQ/MOC 

Compatibility: 
Organization  Alignment 

2.0 

2.0 

Organization/Guidance : 
MHQ/MOC 

Compatibility:  Process 
Alignment 

3.0 

3.0 

System  Performance: 
System  Management 
and  Security:  System 
Management  and 

Security 

1.7 

2.5 

2.0 

System  Performance: 
Automation:  Alerts 

2.3 

2.3 

2.5 

3.0 

3.0 

1.3 

2.3 

System  Performance: 
Automation:  Information 
Processing 

3.0 

3.0 

System  Performance: 
Operations  Support: 
Standards  and 

Guidelines 

2.0 

2.0 

System  Performance: 
Operations  Support: 
System  Utility 

3.0 

2.0 

3.0 

3.0 

3.0 

2.0 

3.0 

3.0 

2.3 

2.6 

System  Performance: 
Technical  Performance: 
Information  Processing 

2.0 

2.0 
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Sub -Sub -Area 

CMA 

E-MIO  Wireless 

FASTC2AP 

Global  Trader 

Google  Earth 

LiNX 

MAGNET 

MASTER 

MDA  DS  COI 

MIDAS 

PANDA 

Tripwire 

All  MDA  Teeh 

Average 

System  Performanee: 
Teehnieal  Performanee: 
Information  Retrieval 

2.3 

2.0 

3.0 

2.5 

2.2 

3.0 

2.5 

2.0 

2.3 

2.3 

System  Performanee: 
Teehnieal  Performanee: 
Information  Sharing 

2.2 

3.0 

2.0 

1.5 

3.0 

2.0 

2.0 

3.0 

1.8 

2.2 

System  Performanee: 
Teehnieal  Performanee: 
Interoperability 

2.0 

2.0 

2.0 

2.0 

System  Performanee: 
Warfighter  Aeeeptanee: 
Human- System 

Interaetion 

2.5 

2.0 

2.3 

3.0 

2.0 

2.0 

3.0 

2.0 

2.4 

System  Performanee: 
Warfighter  Aeeeptanee: 
System  Training 

2.5 

2.4 

2.0 

2.0 

2.0 

2.3 

System  Performanee: 
Warfighter  Aeeeptanee: 
System  Usage 

3.0 

3.0 

2.5 

2.8 

System  Performanee: 
Warfighter  Aeeeptanee: 
System  Utility 

2.8 

2.0 

2.5 

2.0 

2.5 

2.0 

3.0 

3.0 

2.6 

System  Supportability 
and  Readiness:  System 
Supportability  and 
Readiness 

2.0 

2.0 

Average 

2.4 

2.1 

2.4 

2.2 

2.5 

2.4 

2.0 

2.3 

2.0 

2.6 

2.9 

2.9 

2.1 

2.4 

To  make  a  more  foeused  assessment  of  operational  utility  of  systems,  we  eomputed  the 
average  sum  of  seores  for  strengths  and  eoneerns^  about  systems.  Several  MDA 
teehnologies  appear  to  have  partieularly  high  or  low  operational  utility,  under  this 

analysis  (see 

Table  11).  FASTC2AP,  Global  Trader,  PANDA,  Tripwire,  and  CMA  reeeived  high 
marks  for  operational  utility.  MDA  DS  COI,  E-MIO  Wireless,  and  MAGNET  reeeived 
low  marks  for  operational  utility. 


Table  11.  Average  sum  of  concern  and  strength  scores  per  technology. 


*  The  value  here  is  eomputed  as  the  sum  of  all  seores  for  eoneems  (eaeh  valued  at  2  points)  and  strengths  (3 
points)  divided  by  the  total  frequeney  of  these  seores  for  items  in  these  eategories:  System  Performanee 
eategories  for  Operations  Support:  System  Utility,  and  Warfighter  Aeeeptanee:  System  Utility.  Note  that 
seores  for  potential  defieieneies  were  exeluded  from  this  analysis,  as  were  seores  in  all  other  assessment 
eategories. 
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MDA  Teehnology 

Avg  Sum  of  Seores 

FASTC2AP 

3.0 

Global  Trader 

3.0 

PANDA 

3.0 

Tripwire 

3.0 

CMA 

2.8 

MASTER 

2.7 

Google  Earth 

2.5 

EiNX 

2.3 

MDA  DS  COI 

2.0 

E-MIO  Wireless 

1.5 

MAGNET 

n.a. 

An  additional  analysis  was  conducted  to  help  the  reader  assess  the  impact  of  specific 
systems  regarding  fulfillment  of  MD A  eapability  requirements  (see 
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Table  12).  Spiral-1  technologies  must  help  the  Navy  to  fulfill  several  MDA  capabilities: 
monitor,  collect,  fuse,  analyze,  and  disseminate.  One  technology  -  E-MIO  Wireless  -  has 
a  low  assessment  score  (above,  driven  by  connectivity  deficiencies)  and  it  addresses  only 
one  MDA  capability.  This  should  raise  concern  about  the  value  of  this  technology,  and 
focus  investment  (or  disinvestment)  decisions  on  it. 

Low  assessment  scores  (of  2.0),  in  Table  11,  are  also  a  concern  with  respect  to  MAGNET 
and  MDA  DS  COT  However,  these  technologies  address  multiple  MDA  capabilities. 
Strengths  in  one  area  may  compensate  for  concerns  in  others. 

Note  that  all  MDA  capabilities  are  addressed  by  more  than  one  technology.  Thus,  the 
issues  identified  with  respect  to  any  one  technology  do  not  necessarily  indicate  a 
capability  gap.  More  specifically,  three  SP-1  technologies  (CMA,  MAGNET,  and 
Tripwire)  address  all  of  these  capabilities.  Three  technologies  (Global  Trader,  DS/COI, 
and  EASTC2AP)  address  most  capabilities.  Three  technologies  address  only  one 
capability  (E-MIO:  Collect;  Google  Earth:  Disseminate;  and  LiNX:  Collect). 

A  finer-grained  understanding  of  the  implications  of  findings  on  MDA  capabilities  can  be 
had  by  considering  the  mapping  of  technologies  to  the  performance  thresholds  that  each 
technology  must  achieve  (see  section  16.0). 
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Table  12.  Mapping  of  SP-1  Technologies  to  MDA  Capabilities, 


Capability 

CM 

A 

E-MIO 

Wireles 

s 

Globa 

1 

Trade 

r 

Googl 

e 

Eartb 

LIN 

X 

MAGNE 

T 

Data 

Sbarin 

gCOI 

Tripwir 

e 

FASTC2A 

P 

Assessmen 
t  score 

2.4 

2.1 

2.2 

2.5 

2.4 

2.0 

2.0 

2.9 

2.4 

Monitor 

X 

X 

X 

X 

X 

X 

Collect 

X 

X 

X 

X 

X 

X 

X 

X 

Fuse 

X 

X 

X 

X 

Analyze 

X 

X 

X 

X 

X 

Disseminat 

e 

X 

X 

X 

X 

X 

X 

X 

5.0  MDA  Capabilities  Recommendations 

In  this  section,  we  draw  recommendations  from  the  findings,  discussed  in  previous 
sections.  These  recommendations  concern  fielding  and  support,  technology  usability, 
technology  functionality,  data  quality,  training,  organizational  interoperability,  and  future 
MDA  assessment  activity.  They  are  organized  into  three  categories  of  the  assessment 
framework,  and  an  additional  category  concerning  the  programmatics  of  MDA 
assessment. 


5.1  System  Supportability  and  Readiness 

1 .  Strong  technologies:  PEO  C4I  and  OPNAV  should  press  forward  with 
development  and  fielding  of  several  technologies  that  received  high  marks  for 
operational  utility:  FASTC2AP,  Global  Trader,  PANDA,  Tripwire,  and  CMA^. 

2.  Weaker  technologies:  OPNAV  should  evaluate  the  expressed  concerns,  deficits, 
and  return  on  investments  for  several  technologies  that  received  low  marks  for 
operational  utility:  MDA  DS  COI,  E-MIO  Wireless,  and  MAGNET. 

3.  Fielding  sites:  OPNAV  should  consider  placing  technologies  in  reach-back 
facilities  (e.g.,  at  ONI  rather  than  at  NAVCENT)  with  robust  technical  support 
and  operator  competence  for  those  technologies  that  have  high  utility  but  low 
accuracy,  reliability,  or  usability. 

4.  Fielding/support  process:  OPNAV  and  PEO  C4I  should  ensure  and  advertise 
competent  fielding  and  support  for  MDA  technologies.  This  may  involve 


^  This  list  was  arrived  at  by  computing  the  average  weighted  sum  of  scores  for  items  in  two  categories 
concerning  operational  utility:  System  Performance  categories  for  Operations  Support:  System  Utility,  and 
Warfighter  Acceptance:  System  Utility.  Scores  for  potential  deficiencies  were  excluded  from  this  analysis. 
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educating  these  organizations  about  the  processes  for  surveying  infrastructure 
(e.g.,  the  adequacy  of  power  supplies  and  server  space),  configuring  or 
customizing  technologies  to  local  needs,  specifying  technical  support 
requirements,  fulfilling  them,  and  identifying  and  addressing  shortfalls. 

5.2  System  Performance 

5.  Usability:  OPNAV  and  PEO  C4I  should  systematically  analyze  the  usability  of 
MDA  technologies  -  specifically  programmable  agents,  maps,  search  features, 
graphs,  alerts,  and  briefing  products  -  to  ensure  that  (1)  the  cost  in  errors  and 
response  time  is  estimated,  (2)  design  modifications  are  prioritized  accordingly 
and  are  funded,  and  (3)  training  enables  operators  to  work  around  persistent 
usability  problems.  In  MDA  assessment  events,  CMA,  MASTER,  EASTC2AP, 
MDA  DS  COI,  and  E-MIO  all  had  usability  issues  that  raised  concern. 

6.  Redundant  functionality:  OPNAV  and  PEO  C4I  should  evaluate  reports  of 
redundancy  of  MDA  technologies  with  each  other  and  with  existing  systems. 
Eielding  decisions,  TTPs,  and  training  should  resolve  these  redundancies. 

7.  Baseline  specification  functions:  OPNAV  and  PEO  C4I  should  ensure  that 
technologies  with  alert/alarm  capabilities  enable  the  user  to  specify  baseline 
behaviors  for  traffic  in  different  regions.  Without  this  function,  alerts  lose  much 
of  their  value. 

8.  Data  sources:  OPNAV  should  identify,  develop,  and  link  to  critical  data  sources. 
Eor  example,  data  sharing  agreements  are  required  to  provide  data  concerning 
U.S.  persons. 

9.  Data  source  education:  OPNAV  and  PEO  C4I  should  ensure  that  operators  of 
new  systems  understand  which  data  sources  feed  those  systems,  the  reliability  of 
those  sources,  which  expected  data  sources  do  not  feed  those  systems,  and  the 
implications  of  both  for  analyzing  results.  These  explanations  should  identify  the 
reason  for  data  gaps,  e.g.,  lack  of  institutional  or  international  agreements  to 
access  data,  lack  of  a  connection  to  the  data  source,  lack  of  data  replication,  or 
data  loss  during  system  failures.  In  MDA  assessment  events,  unreliable  sources  or 
systems  produced  gaps  in  CMA  track  coverage;  lack  of  data  replication  produced 
incomplete  data  in  Global  Trader;  connectivity  issues  hindered  use  of  E-MIO 
(e.g.,  below  decks);  and  MAGNET  data  were  sparse  per  policies  about  handling 
data  concerning  U.S.  persons. 

10.  Common  Data:  OPNAV  should  analyze  and  resolve  significant  differences  in 
data  sets  between  users  who  must  coordinate  their  activities. 
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1 1 .  Training:  OPNAV  should  implement  a  dedieated  eourse  of  instruetion 
eonoerning  MDA  TTPs,  SOPs,  and  the  role  of  MDA  teehnologies  in  them,  and 
inelude  it  in  the  Navy  Training  System  Plan  (NTSP).  Ideally,  this  training  will  be 
eustomized  to  loeal  missions  and  eonditions. 

5 . 3  Organization/ Guidance 

12.  Process  interoperability:  OPNAV  should  evaluate  and  revise  the  emerging 
proeess  arehiteetures  for  MHQ  with  MOC,  ONI,  and  other  organizations  to  ensure 
that  they  support  MDA  tasks,  and  that  they  ean  be  eustomized  to  the  eonditions  of 
MDA  work  in  the  varied  Navy  Areas  of  Operations.  This  analysis  may  entail 
developing  MDA  use  eases  or  seenarios  against  whieh  to  test  proeess 
arehiteetures,  and  these  use  eases  ean  serve  double  duty  as  exereise  and  training 
seenarios. 

13.  Information  flow  impedance:  OPNAV  and  PEO  C4I  should  assess  the  impaet  of 
new  teehnologies,  proeedures,  and  organizational  struetures  on  the  rate  of 
information  flow  between  organizations,  and  between  elements  of  organizations. 
New  teehnologies  have  the  potential  to  raise  the  information  output  of  some 
organizations  (e.g.,  ONI)  well  above  the  input  and  proeessing  eapaeities  of  the 
organizations  they  support  (e.g.,  NAVCENT). 

5.4  Assessment 

14.  MDA  operational  baseline:  To  estimate  the  return  on  investment  in  MDA 
systems  requires  that  we  define  the  baseline  of  MDA  eapabilities  (e.g.,  the 
number  of  VOIs  developed  and  traeked  per  unit  time).  This  baseline  might  be 
estimated  by  operational  experts.  However,  more  reliable  data  will  arise  from 
direet  observation  of  the  effeetiveness  of  eurrent,  fielded  teehnologies  for  MDA 
missions.  We  reeommend  that  observational  data  be  gathered  eoneerning  the 
eurrent  MDA  baseline. 

15.  MDA  system  alternatives:  The  return  on  investment  for  MDA  assessment  may  be 
inereased  if  experimentation  and  observation  address  some  promising  systems 
outside  the  Spiral- 1  suite.  Sueh  systems  inelude  CMMA,  NEPTUNE,  GAEE-Eite, 
Palaemon,  PANDA,  and  Sea  Wateh.  Evaluation  of  sueh  systems  is  essentially  a 
high  risk/high  return  investment  in  the  portfolio  of  MDA  assessment  aetivities. 

16.  MDA  assessment  scenarios:  OPNAV  and  PEO  C4I  should  develop  exereises  that 
train  and  test  MDA  eapabilities  (teehnologies,  TTPs,  organizations,  ete.),  with  a 
partieular  foeus  on  handling  realistie  numbers  of  white  vessels.  Design  these 
exereises  to  answer  at  least  these  questions:  Can  we  develop  a  VOI  and  identify  it 
among  many  vessels  almost  identieal  to  it?  Can  we  eommunieate  seeurely  and 
eonfidentially  with  a  single  ship  among  many  ships  in  an  area?  Can  we  eoordinate 
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our  actions  in  a  crowded  field  of  internationally  flagged  white  vessels?  Can  we 
manage  and  address  eosts  of  delay  to  eommereial  shipping  (estimated  by 
NORTHCOM  at  $  10,000/hour)  and  erroneous  aetions  (e.g.,  damage  or 
destruetion  of  internationally  flagged  vessels)? 

17.  MDA  process  modeling:  OPNAV  should  ensure  that  future  assessments  use 
measures  that  support  eomputational,  “what  if’  modeling  of  the  impaet  of  new 
teehnologies,  proeesses,  manning,  and  organizational  struetures.  Such  measures 
should  represent  the  speed-aeeuraey  tradeoff  curve  for  analysis,  deeision  making, 
and  aetion  (throughput)  given  varied  missions,  staff  size  and  eompetency,  and 
related  factors.  By  reusing  these  measurements  in  models,  the  Navy  will  multiply 
the  answers  it  can  extract  from  scant  assessment  data,  and  thus  inerease  its  return 
on  every  dollar  spent  on  assessment. 
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6.0  System  Performance  Results 

The  assessment  area  “system  performance”  concerns  the  performance  of  MDA 
technologies,  including  Spiral- 1  technologies.  The  components  of  this  assessment  area 
concern  the  technical  performance  of  systems,  the  support  they  provide  to  operations, 
warfighter  acceptance  of  the  systems,  automation  of  important  functions,  and  system 
management  and  security. 

The  key  data  concerning  the  System  Performance  assessment  area  are  presented  in  Table 
13.  This  table  (like  those  in  subsequent  sections)  presents  findings  sorted  by  assessment 
area,  technology,  and  assessment  score,  in  that  order.  The  Comments  column  concisely 
summarizes  the  Item  relevant  to  the  Area.  The  Technology  and  Venue  columns  indicate 
the  object  and  source  of  the  item,  respectively.  Note  that  any  one  Item  could  be  coded  in 
multiple  Assessment  Areas  or  concern  multiple  Technologies.  Thus,  some  items  are 
repeated  in  this  table. 


Table  13,  System  Performance  Results. 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

220 

System 
Management 
and  Security: 
System 
Management 
and  Security 

Spiral- 1  systems  management  was  tested 
during  FAIRGAME.  No  reliability  or 
supportability  issues  were  observed.  The 
following  maintenance  issues  were  observed: 

technical 

support 

3 

All  MDA 

Tech 

VBSS 

223 

System 
Management 
and  Security: 
System 
Management 
and  Security 

ONI  expresses  concern  about  integration  and 
accreditation. 

technical 

support 

2 

All  MDA 

Tech 

VBSS 

221 

System 
Management 
and  Security: 
System 
Management 
and  Security 

The  NMIC  CMA  system  suffered  an  unknown 
failure  that  rendered  it  unusable.  An  attempt 
by  NRL  to  troubleshoot  via  remote  access  was 
unsuccessful.  A  technician  arrived  on-scene 
to  conduct  on-site  troubleshooting  and 
discovered  a  faulty  hardware  router  and 
replaced  it  five  hours  later. 

technical 

support 

2 

CMA 

TWOS 

222 

System 
Management 
and  Security: 
System 
Management 
and  Security 

A  potential  issue  is  that  the  primary  means  of 
troubleshooting  a  CMA  problem  is  for  NRL  to 
access  the  local  system  via  remote  web  access. 
If  the  problem  cannot  be  resolved  remotely, 
NRL  must  dispatch  a  technician  to  the  node. 
COMUSNAVCENT  however,  has  a 
permanent  SPAWAR  Fleet  Systems 

Engineering  team  representative  onsite  to 
resolve  CMA  hardware  issues. 

technical 

support 

2 

CMA 

QRA 

64 

System 
Management 
and  Security: 

CMA:  Users  sometimes  lost  track  of  the 
original  security  classification  of  the 
information  they  wished  to  disseminate.  This 

opsec 

1 

CMA 

QRA 

31 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

System 
Management 
and  Security 

increased  the  likelihood  of  a  security  violation 
as  a  result  of  passing  classified  information  on 
the  wrong  domain. 

271 

Automation: 

Alerts 

NAVCENT  staff  see  value  in  technologies 
that  trigger  alerts  concerning  specific  tracks. 

operational 

utility 

2 

All  MDA 

Tech 

Workflow, 

PEW 

204 

Automation: 

Alerts 

The  ability  to  capture  and  store 
baseline/normal  maritime  movement  patterns 
was  not  observed.  Spiral- 1  tools  did  not  alert 
users  to  deviations  from  normal  route  or 
behavior  patterns. 

establishing 

baselines 

1 

All  MDA 

Tech 

VBSS 

208 

Automation: 

Alerts 

Spiral- 1  did  not  automatically  establish  or 
display  threat  assignments  based  upon  a  user- 
defined  alert. 

features 

1 

All  MDA 

Tech 

QRA 

3 

Automation: 

Alerts 

The  majority  of  users  reported  liking  the 
ability  that  CMA  gives  them  to  leave  an  alert 
in  a  vessel  file  that  can  be  used  by  other 
analysts.  Users  also  reported  being  able  to  rely 
on  pre-defined  alerts  to  stay  informed  about 
vessels  of  interest.  Regarding  the  timely 
attribute,  users  reported  that  query  functions 
saved  significant  time.  Specifically,  it  was 
helpful  to  be  able  to  filter  by  location,  time, 
vessel  name  and  vessel  attributes. 

features 

3 

CMA 

QRA 

197 

Automation: 

Alerts 

CMA:  Thirty-seven  percent  of  users  "agreed," 
26%  "strongly  agreed,"  a32%  indicated 
"N/A,"  and  5%  "strongly  disagreed"  they  liked 
the  ability  CMA  gives  to  leave  an  "alert"  in  a 
vessel  file  that  can  be  used  by  other  analysts. 
Forty-two  percent  of  users  indicated  "N/A," 

26%  "agreed,"  21%  indicated  "disagree,"  and 
11%  indicated  "strongly  agree"  that  they  could 
count  on  the  pre-defined  alerts  in  CMA  to 
detect  conditions  of  interest  and  that  CMA 
pre-defined  alerts  helped  them  stay  informed 
about  Vols.  Forty-two  percent  of  users 
indicated  "N/A,"  32%  "agreed,"  11% 
"disagreed,"  and  16%  strongly  agreed  CMA 
pre-defined  alerts  were  understandable. 

operational 

utility 

3 

CMA 

TWOS 

241 

Automation: 

Alerts 

It  was  easy  for  CMA  users  to  associate  tipper/ 
Intel  information  with  the  correct  VOL  CMA 
showed  recent  alerts  on  the  vessel  and  made  it 
easy  to  see  the  tippers. 

usability 

3 

CMA 

TRRLOE 

36 

Automation: 

Alerts 

Suggestions  for  Improvement  and  Useful 
Features.  Both  CMA  users  experienced 
difficulties  using  the  alerting  system  as  well  as 
the  hyper  graph.  Participants  noted  that  when 
searching  on  CMA  using  a  partial  search,  the 
system  did  not  always  pull  up  all  tracks 
containing  a  portion  of  the  name.  One  user 
recommended  adding  a  feature  to  provide  the 
capability  to  use  range  and  bearing  from  a 
point  in  a  circle  to  form  a  search  area.  This 
user  also  suggested  including  the  ability  to  use 
a  line  or  border  to  search  for  any  track 
crossing  a  line  rather  than  a  box  as  is  done  in 
the  current  system. 

usability 

2 

CMA 

QRA 

32 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

173 

Automation: 

Alerts 

CMA  Alerts:  Was  unable  to  set  up  alerts  and 
trip  wires  to  notify  of  transit.  Information  on 
cargo  and  individuals  on  the  vessel  was 
limited. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

VBSS 

211 

Automation: 

Alerts 

Pre-defined  alerts  were  not  representative  of 
current  operational  needs.  CMA  is  not 
capable  of  providing  user-defined  alerts. 

operational 

utility 

2 

CMA 

TWOS 

214 

Automation: 

Alerts 

CMA  -  Three  of  four  users  during 

FAIRGAME  did  not  think  the  alerting  system 
in  CMA  was  easy  to  understand.  ("I  might  not 
have  been  trained  well  enough  on  using  it  hut 

I  did  not  find  the  alerting  system  especially 
helpful,  save  for  the  ability  to  save  watch  areas 
which  is  an  easy  system.")  Regarding  whether 
the  alerting  system  in  CMA  was  useful  and 
relevant,  users  were  either  neutral  or 
disagreed,  e.g.,  "To  set  up  the  trips  was  not 
easy.  Establishing  the  bounding  boxes  and 
setting  up  the  search  parameters  was  easy  hut 
needed  further  instruction  on  alert  set  up." 
(FAIRGAME) 

usability 

2 

CMA 

VBSS 

206 

Automation: 

Alerts 

Spiral- 1  tools  did  not  provide  additional 
capability  to  establish  baseline  normal  civil 
maritime  operations  worldwide  and  threat 
assessment  criteria.  CMA  and  FASTC2AP 
could  alert  based  upon  a  geographic 
point/area/proximity,  hut  did  not  support  alerts 
employing  algorithms  based  upon  baseline 
maritime  operations. 

establishing 

baselines 

1 

CMA 

QRA 

69 

Automation: 

Alerts 

FASTC2AP  provides  a  basic  user-defined 
anomaly  detection  capability. 

operational 

utility 

3 

FASTC2AP 

QRA 

117 

Automation: 

Alerts 

Usability.  The  operator  felt  that  FASTC2AP 
was  difficult  to  learn  resulting  in  the  user 
having  to  try  the  agents  multiple  times  in  order 
to  understand  what  each  was  providing  in  the 
way  of  results.  The  operator  stated  it  was 
fairly  easy  to  create  agents,  however 
understanding  the  results  is  difficult.  It  was 
easy  to  compose  agents  and  to  build  alerts  in 
order  to  fill  RFIs,  although  the  system  could 
not  provide  the  details  required  for  some  of 
the  RFIs. 

usability 

3 

FASTC2AP 

VBSS 

209 

Automation: 

Alerts 

FASTC2AP  alerted  the  user,  per  user-defined 
alerts,  when  a  vessel  meets  the  alert 
thresholds.  The  user  then  conducted  a  threat 
analysis  on  the  vessel.  The  Tripwire  system 
will  alert  the  user  that  there  is  a  data  element 
that  meets  search  criteria. 

alerts 

3 

FASTC2AP 

QRA 

178 

Automation: 

Alerts 

FASTC2AP  has  predefined  alert  agents  and 
the  facility  for  operators  to  define  their  own 
alerts.  The  agents  were  useful  but 
understanding  some  of  the  agent  results  was 
difficult.  This  was  due  to  agent  templates 
being  difficult  to  understand.  Setting  up  new 
agents  is  difficult.  The  system  is  non- 

usability 

2 

FASTC2AP 

TRRLOE 

33 
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ID 
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Score 

Technology 
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intuitive. 

212 

Automation: 

Alerts 

Change  FASTC2AP  alerts  so  SCONUM  is 

Not  Required.  FASTC2AP  has  a  number  of 
pre-defined  alert  functions.  Their  use  requires 
an  ONI-assigned  SCONUM,  which  when 
combined  with  a  ship’s  name  and  its  position 
becomes  classified.  The  SCONUM  also  may 
not  be  available  to  coalition  forces.  This 
makes  these  alerts  of  limited  use  and  could 
cause  future  security  concerns.  It  is 
recommended  that  the  alerts  be  rewritten  to 
require  ship  names  only  or  the  Lloyd’s 
identification  number. 

usability 

2 

FASTC2AP 

TRRLOE 

207 

Automation: 

Alerts 

Spiral- 1  tools  did  not  provide  additional 
capability  to  establish  baseline  normal  civil 
maritime  operations  worldwide  and  threat 
assessment  criteria.  CMA  and  FASTC2AP 
could  alert  based  upon  a  geographic 
point/area/proximity,  but  did  not  support  alerts 
employing  algorithms  based  upon  baseline 
maritime  operations. 

establishing 

baselines 

1 

FASTC2AP 

QRA 

199 

Automation: 

Alerts 

MASTER:  Respondents  agreed  that 
master’s  pre-defined  alerting  capabilities 
are  superior  to  current  methods  and  that 
master’s  user-defined  alerts  supported 
accurate  detection  of  conditions  of  interest. 

This  capability  was  considered  to  be  an 
excellent  feature,  able  to  respond  to  the 
dynamic  maritime  threat  environment, 
assuming  that  users  provide  accurate  rules  for 
alerts. 

operational 

utility 

3 

MASTER 

TWOS 

200 

Automation: 

Alerts 

MASTER:  One  respondent  wanted  a  way  to 
see  alerts  for  his  rules  only,  not  everyone’s 
rules;  another  respondent  noted  that 

MASTER,  unlike  Sealink,  cannot  remember  a 
search  parameter,  requiring  the  same  search  to 
be  built  each  day,  which  is  time  consuming. 

features 

3 

MASTER 

TWOS 

63 

Automation: 

Alerts 

MASTER:  MASTER  provides  a  single  source 
of  tools  which  are  superior  to  legacy,  but  users 
would  prefer  the  two  systems  (MASTER  and 
legacy)  to  interact.  Users  typically  identified 
the  correct  tracks  in  their  assigned  scenarios, 
but  various  usability  issues  were  observed. 

Some  users  questioned  the  completeness  or 
accuracy  of  the  information  in  MASTER’S 
database.  Several  users  used  SeaLink  to  find 
information  on  a  port  and  to  correlate  old  ship 
names  with  new  ship  names.  One  user  utilized 
the  PMIC  web  page  in  attempting  to 
determine  the  destination  for  a  particular  ship. 
Two  users  preferred  ASA  over  MASTER 
because  it  is  both  faster  and  easier  to  use,  not 
because  it  is  more  accurate  than  MASTER. 
Respondents  agreed  that  they  could  count  on 
the  pre-defined  alerts  in  MASTER  to  detect 
conditions  of  interest.  Because  alerts  are  not 
easy  to  set  up  using  alert  management  and 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

MASTER 

TWOS 

34 
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ID 
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Technology 

Venue 

alert  channels,  one  respondent  recommended 
that  alerts  be  based  on  completed  searches  and 
that  an  option  be  provided  to  save  that  search 
as  an  alert. 

176 

Automation: 

Alerts 

Tripwire,  Alert:  Simply  unable  to  use.  With 
more  training  am  sure  that  too  could  be 
mastered 

usability 

2 

MASTER 

VBSS 

45 

Automation: 

Alerts 

PANDA:  Regarding  predicting  deviation, 
most  users  (85%)  responded  that  detecting 
deviation  is  a  useful  capability  and  all  users 
responded  that  alerting  for  deviation  is  a 
useful  capability.  Most  users  responded  that 
the  capability  of  classifying  alerts  into 
categories  (86%)  and  that  PANDA’ s 
classification  of  alerts  (86%)  were  useful.  User 
responses  were  mixed  about  whether 
panda’s  explanation  for  the  deviation  was 
useful.  Information  that  users  would 
communicate  to  others  about  deviations 
included  %  likelihood  a  vessel  would  deviate, 
context  information  such  as  ship 
parts/ spares/repair,  and  crew  change/illness 
(currently  not  available  in  PANDA),  other 
vessels  in  the  area,  whether  a  vessel  maybe  a 
potential  threat  or  information  for  a  task. 

operational 

utility 

3 

PANDA 

TW08 

201 

Automation: 

Alerts 

PANDA:  Regarding  deviation:  All  users 
responded  that  PANDA  was  effective  at 
alerting  for  deviations  and  that  PANDA’s 
explanation  for  the  deviation  was  reasonable 
and  made  sense.  Most  users  indicated  that  it 
was  easy  to  understand  the  explanation  for  the 
deviation  (67%  and  83%  strongly  agreed; 
others  were  neutral).  Regarding  prediction: 
Sixty-seven  percent  of  users  (67%)  responded 
that  PANDA  was  effective  at  representing 
predicted  vessel  behavior  (others  were  neutral 
and  disagreed). 

operational 

utility 

3 

PANDA 

TW08 

203 

Automation: 

Alerts 

PANDA:  Deviation:  All  users  indicated  that 
the  following  capabilities  were  useful  - 
classifying  alerts,  allowing  users  to  update  the 
alert  classification  and  provide  feedback.  Most 
users  indicated  that  PANDA  was  effective  at 
detecting  deviations  (85%,  others  =  slightly 
effective  and  n/a),  and  all  users  indicated 
PANDA  was  effective  at  classifying  alerts.  All 
users  also  indicated  that  it  was  easy  to  change 
the  classification  for  the  deviation  provided  by 
PANDA  and  to  explain  the  change. 
Identification  of  deviations  in  a  similar  area 
was  deemed  to  be  helpful  (79%  of  users). 

Most  users  (93%)  agreed  that  the  ability  to 
associate  correlated  deviations  would  be  a 
useful  feature  to  have  in  PANDA. 

operational 

utility 

3 

PANDA 

Workflow, 

PEW 

218 

Automation: 

Alerts 

PANDA:  All  12  users  indicated  PANDA 
would  have  a  positive  effect  on  their  situation 
awareness.  Users  pointed  to  a  variety  of 
PANDA  features  that  would  support  improved 

operational 

utility 

3 

PANDA 

QRA 

35 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

SA,  including  an  increase  in  understanding  of 
normalcy  and  anomalous  vessel  behavior, 
alerting  to  behavior  that  would  otherwise  be 
missed,  prediction  related  to  ships  inbound  to 

US  ports,  understanding  of  vessel  history,  and 
potential  support  for  "proactive"  analysis. 

210 

Automation: 

Alerts 

FASTC2AP  alerted  the  user,  per  user-defined 
alerts,  when  a  vessel  meets  the  alert 
thresholds.  The  user  then  conducted  a  threat 
analysis  on  the  vessel.  The  Tripwire  system 
will  alert  the  user  that  there  is  a  data  element 
that  meets  search  criteria. 

alerts 

3 

Tripwire 

QRA 

90 

Automation: 

Information 

Processing 

Respondents  mostly  agreed  that  MASTER'S 
capability  to  automatically  acquire  data  on 
vessels  of  interest  is  superior  to  current  data 
acquisition  processes.  MASTER’S  track 
analysis  and  graphing  capabilities  were  among 
its  strengths,  but  its  archive  database  was  often 
slow  to  respond  to  queries.  MASTER  was 
easy  to  use  when  creating  new  vessel  tracks. 

automated 
data  analysis 

3 

MASTER 

TW08 

87 

Operations 
Support: 
Standards  and 
Guidelines 

Current  Standard  Operating  Procedures  (SOP) 
and  TTP  do  not  address  the  new  MDA  Spiral- 
1  tools.  OPNAV  N3/N5  has  developed  an 
outstanding  draft  MDA  SOP  that,  with  further 
refinement  and  extensive  Fleet  feedback  will 
provide  the  groundwork  for  all  MDA  nodes  to 
employ  the  new  technologies  to  their  fullest 
extent.  The  SOP  in  its  current  version  will 
assist  the  users  in  navigating  through  the 
different  technologies,  but  requires  feedback. 

SOP  and 

TTPs 

2 

All  MDA 

Tech 

QRA 

259 

Operations 

Support: 

System 

Utility 

MIFCPAC  will  require  MDA  tools  to 
discriminate  between  different  types  or 
activities  of  white  vessels,  e.g.,  US  vs.  foreign 
vessels  in  fisheries. 

features  for 
vessel  type 
discrimination 

3 

All  MDA 

Tech 

Workflow, 

PEW 

270 

Operations 

Support: 

System 

Utility 

NAVCENT  leadership  views  positively  the 
Spiral  2  initiative  to  combine  the  shore-based 
radars  of  many  nations  with  AIS  data.  This 
capability  would  benefit  operations  and 
strengthen  partnerships. 

political 
utility  of  AIS 
data  sharing 

3 

All  MDA 

Tech 

VBSS 

269 

Operations 

Support: 

System 

Utility 

NAVCENT  is  concerned  about  the  relevance 
of  the  technology  effort  to  primary  missions. 

operational 

utility 

2 

All  MDA 

Tech 

VBSS 

272 

Operations 

Support: 

System 

Utility 

NAVCENT  staff  see  little  value  in 
technologies  for  data  mining  or  fusion  across 
multiple  sources. 

operational 
utility  of  data 
mining 

2 

All  MDA 

Tech 

Workflow, 

PEW 

280 

Operations 

Support: 

System 

Utility 

ONI  expresses  concern  about  lack  of 
functionality  for  data  analysis  and  event 
prediction. 

features  for 
analysis 

2 

All  MDA 

Tech 

VBSS 

286 

Operations 

Support: 

System 

Utility 

CINCPACFLEET  has  only  one  OPPLAN  on 
which  it  needs  to  track  white  shipping,  so  the 
capability  will  be  important  hut  rarely  used. 

operational 

utility 

2 

All  MDA 

Tech 

Workflow, 

PEW 

261 

Operations 

Support: 

MIFCPAC  sees  value  in  CM  A,  Google,  and 
Global  Trader. 

features  for 
vessel  type 

3 

CMA 

Workflow, 

PEW 

36 
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System 

Utility 

discrimination 

265 

Operations 

Support: 

System 

Utility 

ONI  sees  value  in  several  tools:  CMA, 
TRIPWIRE,  TAANDEM,  FASTCAP,  and 
EMIO  wireless. 

operational 

utility 

3 

CMA 

Workflow, 

PEW 

257 

Operations 

Support: 

System 

Utility 

MIO  situation  awareness  was  maintained  at 
rear  units. 

operational 
utility  for 

MIO  SA 

3 

E-MIO 

Wireless 

Workflow, 

PEW 

266 

Operations 

Support: 

System 

Utility 

ONI  is  sees  value  in  several  tools:  CMA, 
TRIPWIRE,  TAANDEM,  FASTCAP,  and 
EMIO  wireless. 

operational 

utility 

3 

E-MIO 

Wireless 

Workflow, 

PEW 

84 

Operations 

Support: 

System 

Utility 

Tactical  EMIO  System  (TES)  -  While 
mobility  of  the  Tactical  EMIO  Device  (TED) 
was  a  clear  advantage,  enabling  the  collection 
of  data  from  multiple  locations  within  the 
vessel,  one  limitation  noted  was  that  the  TED 
must  be  within  the  vicinity  of  the  Tactical 

EMIO  Maritime  PC  (TEMP)  to  download  the 
data  captured.  Although  wireless,  the  TED 
devices  were  required  to  be  in  the  vicinity  of 
the  TEMP  in  order  to  download  data  captured 
during  the  boarding.  The  radio  frequency  (RF) 
signals  were  not  strong  enough  to  transmit 
data  when  team  members  were  below  decks. 

The  Maritime  BGAN  EMIO  Terminal 
(MBET)  device,  in  turn,  failed  to  transfer  data 
due  to  environmental  issues  and  weak  RF 

Signal  range  of  the  commercial  satellite. 
Contractors  eventually  departed  the  target 
vessel  and  drove  inland  with  the  TEMP  and 
MBET  device  to  acquire  a  stronger  signal.  The 
MBET  link  was  then  acquired  and 
successfully  transmitted  data  from  the  TEMP 
device.  Contractors  asserted  that  the  satellite 
connectivity  will  not  be  a  concern  in  the 
current  AOR.  Also,  the  boarding  officer  was 
not  able  to  demonstrate  the  transfer  of  data  via 
the  Maritime  BGAN  EMIO  Terminal  (MBET) 
due  to  the  satellite  connectivity.  (VBSS 

School) 

connectivity 

1 

E-MIO 

Wireless 

VBSS 

121 

Operations 

Support: 

System 

Utility 

Tactical  EMIO  System  (TES)  -  Although 
wireless,  the  TED  devices  were  required  to  be 
in  the  vicinity  of  the  TEMP  in  order  to 
download  data  captured  during  the  boarding. 
The  radio  frequency  (RF)  signals  were  not 
storing  enough  to  transmit  data  when  team 
members  were  below  decks.  (VBSS  School) 

connectivity 

1 

E-MIO 

Wireless 

QRA 

264 

Operations 

Support: 

System 

Utility 

MIFCPAC  argues  that  FASTC2AP  may  not 
be  "viable"  for  its  uses. 

operational 

utility 

3 

FASTC2AP 

TWOS 

267 

Operations 

Support: 

System 

ONI  is  sees  value  in  several  tools:  CMA, 
TRIPWIRE,  TAANDEM,  FASTCAP,  and 
EMIO  wireless. 

operational 

utility 

3 

FASTC2AP 

Workflow, 

PEW 

37 
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Utility 

262 

Operations 

Support: 

System 

Utility 

MIFCPAC  sees  value  in  CM  A,  Google,  and 
Global  Trader. 

features  for 
vessel  type 
discrimination 

3 

Global 

Trader 

Workflow, 

PEW 

263 

Operations 

Support: 

System 

Utility 

MIFCPAC  sees  value  in  CM  A,  Google,  and 
Global  Trader. 

features  for 
vessel  type 
discrimination 

3 

Google 

Earth 

Workflow, 

PEW 

83 

Operations 

Support: 

System 

Utility 

The  full  potential  use  of  LINX  requires  a  shift 
in  current  NCIS  investigative  processes. 

misc 

2 

LiNX 

VBSS 

97 

Operations 

Support: 

System 

Utility 

MASTER:  Respondents  agreed  they  could 
track  vessels  as  needed.  MASTER  does  a  very 
good  job  of  pulling  vessel  track  history, 
provides  named  data  that  current  systems  do 
not,  and  facilitates  vessel  tracking  with  its 
graphical  display  and  ability  to  compare 
tracks. 

operational 
utility  for 
tracking 

3 

MASTER 

TW08 

111 

Operations 

Support: 

System 

Utility 

Tripwire  Contextual  information  on  vessels  of 
interest  and  persons  of  interest  is  outstanding. 

fusion  of  data 

3 

Tripwire 

VBSS 

268 

Operations 

Support: 

System 

Utility 

ONI  is  sees  value  in  several  tools:  CMA, 
TRIPWIRE,  TAANDEM,  FASTCAP,  and 
EMIO  wireless. 

operational 

utility 

3 

Tripwire 

Workflow, 

PEW 

53 

Technical 

Performance: 

Information 

Processing 

Spiral- 1  tools  did  not  conduct  statistical 
analysis  of  data  gaps  in  order  to  identify 
potential  new  sources  of  information  and  drive 
new  collections. 

automated 
data  analysis 

2 

All  MDA 

Tech 

n.a. 

49 

Technical 

Performance: 

Information 

Retrieval 

Spiral- 1  tools  increased  the  user’s  ability  to 
fuse  vessel,  people,  and  cargo  data  from 
interagency  sources 

fusion  of  data 

3 

All  MDA 

Tech 

QRA 

25 

Technical 

Performance: 

Information 

Retrieval 

Many  of  the  FAIRGAME  users  had 
experience  in  previous  Operational 
Demonstrations  (OD)  as  well  as  in  Trident 
Warrior  and  noted  that  system  information 
reliability  continued  to  be  a  concern. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

All  MDA 

Tech 

TW08 

47 

Technical 

Performance: 

Information 

Retrieval 

The  lack  of  commonality  between  nodes  made 
the  aggregation  of  information  difficult. 

MDA  Spiral- 1  did  not  contain  a  central 
repository  of  worldwide  vessel  movement 
data.  Differences  in  node  metadata  required 
additional  research  on  the  part  of  the  user, 
increasing  the  time  required  to  perform  a 
mission. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

All  MDA 

Tech 

QRA 

77 

Technical 

Performance: 

Information 

Retrieval 

Large  data  latency  was  observed  in  littoral 
regions. 

speed 

2 

All  MDA 

Tech 

TW08 

1 

Technical 

Performance: 

Information 

The  majority  of  users  reported  liking  the 
ability  that  CMA  gives  them  to  leave  an  alert 
in  a  vessel  file  that  can  be  used  by  other 

usability  of 
alerts 

3 

CMA 

EC  EMIO 

38 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

Retrieval 

analysts.  Users  also  reported  being  able  to  rely 
on  pre-defined  alerts  to  stay  informed  about 
vessels  of  interest.  Regarding  the  timely 
attribute,  users  reported  that  query  functions 
saved  significant  time.  Specifically,  it  was 
helpful  to  be  able  to  filter  by  location,  time, 
vessel  name  and  vessel  attributes. 

5 

Technical 

Performance: 

Information 

Retrieval 

CMA  is  viewed  as  enhancing  SA  rapidly. 

Ninety  percent  of  users  either  agreed  or 
strongly  agreed  that  they  could  maintain 
awareness  of  maritime  activity  through  CMA. 
Forty-two  percent  of  users  "agreed,"  26% 
"strongly  agreed,"  16%  "disagreed,"  and  16% 
indicated  "N/A"  that  CMA’s  capability  to 
automatically  acquire  data  on  vessels  of 
interest  from  multiple  databases  is  faster  than 
current  data  acquisition  processes/ systems. 
Forty-seven  percent  of  users  "strongly 
agreed,"  32%  "agreed,"  16%  were  "n/a"  and 

5%  of  users  "disagreed"  that  CMA’s  query 
capabilities  (search  and  advanced  search)  save 
them  considerable  time  in  gathering 
information  on  a  vessel  of  interest  compared 
to  current  system  capabilities.  Fifty-three 
percent  of  users  "strongly  agreed"  and  42% 
"agreed"  that  CMA’s  advanced  search 
capability  to  filter  information  by  location, 
time,  vessel  name,  vessel  attributes,  etc.,  saves 
them  time  compared  with  having  to  do 
multiple  individual  searches. 

usability  of 
search 

3 

CMA 

TWOS 

29 

Technical 

Performance: 

Information 

Retrieval 

Very  little  time  was  required  for  operators  to 
access  information  from  CMA,  typically  1-10 
min.  This  was  with  pre-selected  ships  where 
information  was  known  to  exist. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

3 

CMA 

QRA 

37 

Technical 

Performance: 

Information 

Retrieval 

Participants  thought  conducting  analysis  on  a 
VOI  was  easier  and  much  faster  when  using 
CMA  than  with  their  previous  system. 

speed 

3 

CMA 

TWOS 

109 

Technical 

Performance: 

Information 

Retrieval 

Although  the  data  reliability  issue  cited  by 
users  remains  a  concern,  CMA  is  still  a 
capable  system  and  greatly  enhances  the 
warfighter’s  ability  to  maintain  situational 
awareness. 

operational 
utility  for  SA 

3 

CMA 

QRA 

144 

Technical 

Performance: 

Information 

Retrieval 

Most  users  agreed,  or  strongly  agreed,  CMA 
was  easy  to  learn  and  use,  as  evidenced  by  the 
following:  "With  the  exception  of 
technological  glitches  that  occasionally 
hindered  CMA’s  performance  I  was  able  to 
learn  and  maneuver  the  program  very  quickly 
-  it  probably  took  less  than  two  hours  of 
training  and  a  few  examples  to  understand  all 
the  major  features."  Users  were  mixed  about 
the  helpfulness  of  the  online  help  and  about 
whether  it  was  easy  to  use  all  the  features 
included  in  CMA.  (FAIRGAME) 
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Sixty-eight  percent  of  users  "strongly  agreed" 
and  26%  of  users  "agreed"  that  CMA’s 
capability  to  correlate  vessel,  cargo,  or  people 
information  automatically  and  store  it  in  a 
single  system  makes  my  job  easier  and  saves 
me  time.  Thirty-seven  percent  of  users 
"strongly  agreed,"  37%  of  users  "agreed,"  21% 
of  users  indicated  "N/A"  and  5%  of  users 
"disagreed"  that  CMA’s  hypergraph  display  of 
vessel  relationships  with  other  vessels,  crews, 
etc.,  is  useful  in  rapidly  identifying  related 
information. 
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Sixty-three  percent  of  users  "agreed"  and  34% 
of  users  "strongly  agreed"  that  CMA’s  display 
of  vessel  attributes  is  useful  in  their  analysis  of 
vessels  of  interest.  Sixty-three  percent  of  users 
"agreed,"  26%  of  users  "strongly  agreed,"  16% 
of  users  were  "N/A,"  and  5%  of  users 
"disagreed"  that  the  quality  of  CMA 
automated  track  development  functions 
supports  their  analysis  tasks. 

fusion  of  data 

3 

CMA 

TWOS 

238 

Technical 

Performance: 

Information 

Retrieval 

Both  CMA  users  strongly  agreed  it  was  easy 
to  develop  and  maintain  situation  awareness 
on  a  vessel  of  interest  (VOI),  especially  once 
provided  with  a  watch  list  that  includes  the 
names  of  the  vessels.  CMA  operators 
emphasized  that  it  was  extremely  easy  to  find 
relevant  information  on  a  VOI  as  long  as  one 
knew  the  name  of  the  vessel;  otherwise  it 
would  be  difficult  due  to  the  hyper  graph. 
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Forty-two  percent  of  users  "agreed,"  32%  of 
users  "strongly  agreed,"  and  26%  of  users 
indicated  "N/A"  that  their  ability  to 
individually  identify  and  save  analytic  areas  of 
interest  in  CMA  helps  them  in  their  analytic 
work.  Fifty-eight  percent  of  users  "strongly 
agreed"  and  40%  of  users  "agreed"  that 

CMA’s  ability  to  display  the  track  history  of 
vessels  is  important  to  their  analytic  work. 
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CMA  -  All  four  users,  surveyed  during 
FAIRGAME,  agreed/  strongly  agreed  it  was 
easy  to  search  and  track  a  Vol.  Users  were 
mixed  regarding  whether  they  were  able  to 
process  and  maintain  situation  awareness  on 
more  Vols  then  with  their  previous  system. 
(FAIRGAME) 
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CMA  provided  searchable  data  on  vessel 
location,  cargo,  and  people  but  there  were 
gaps  in  track  data,  a  lack  of  commonality  of 
data  between  nodes,  and  insufficient 
information  on  data  sources. 
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CMA:  On  several  occasions  a  query  of  current 
vessel  position  returned  multiple  vessel 
positions,  each  with  different  metadata  (time, 
cargo,  flag)  attached. 
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Users  noted  a  general  inconsistency  of  the 

CMA  vessel  database  between  nodes  within 
the  same  AOR.  An  analysis  was  performed 
on  the  commonality/consistency  of  track  data 
held  by  the  NAVCENT  and  PACFLT  CMA 
systems  within  a  5  by  10  deg  geographic 
region  in  the  Indian  Ocean.  Although  97%  of 
the  tracks  were  held  by  both  systems, 
significant  track  metadata  differences  were 
observed  in  1 5%  of  the  tracks,  and  track 
history  data  were  only  51%  common  (77% 
common  for  tracks  going  from  the  NAVCENT 
AOR  to  the  PACFLT  AOR,  and  29%  common 
for  tracks  going  in  the  other  direction).  Some 
track  history  data  in  one  site’s  CMA  were  not 
in  the  other  site’s  CMA. 
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Users  consistently  cross  referenced  other  data 
sources  to  validate  query  responses  returned 
by  CMA.  The  user  had  limited  information  on 
information  feeds  into  CMA.  The  users’ 
uncertainty  in  the  completeness  of  the  data 
affected  confidence  in  the  search  results. 
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The  percentage  of  worldwide  cargo  data  fed  to 
CMA  by  Global  Trader  was  not  known.  The 
percentage  of  worldwide  cargo  data  available 
via  Cargo  Link  was  not  known. 
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Users  doubted  the  completeness  of  the  CMA 
database  and  used  other  systems  to  validate 
the  CMA  search  result.  It  is  unclear  to  the 
user  what  data  is  actually  feeding  CMA;  users 
noted  the  need  for  comparable  data  as 
available  with  the  legacy  systems. 
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Users  also  noted  that  certain  information  is 
stripped  from  the  Tripwire  database  before  it 
reaches  the  user,  but  what  is  stripped  is 
unknown. 
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During  FAIRGAME,  users  were  mixed 
(responses  ranged  from  disagree  to  agree) 
regarding  whether  they  were  able  to  develop 
awareness  of  VOIs  faster  with  CMA  "Again, 
definitely  when  there  was  a  good  fused  picture 
or  a  complicated  track  I  needed  to  follow.  I 
imagine  for  most  VOIs  it  won’t  be 
significantly  faster,  though."  All  users 
agreed/strongly  agreed  it  was  easy  to  find 
information  on  a  Vol.  (FAIRGAME) 
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Suggestions  for  Improvement  and  Useful 
Features.  Both  CMA  users  experienced 
difficulties  using  the  alerting  system  as  well  as 
the  hyper  graph.  Participants  noted  that  when 
searching  on  CMA  using  a  partial  search,  the 
system  did  not  always  pull  up  all  tracks 
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containing  a  portion  of  the  name.  One  user 
recommended  adding  a  feature  to  provide  the 
capability  to  use  range  and  bearing  from  a 
point  in  a  circle  to  form  a  search  area.  This 
user  also  suggested  including  the  ability  to  use 
a  line  or  border  to  search  for  any  track 
crossing  a  line  rather  than  a  box  as  is  done  in 
the  current  system. 
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CMA  users  suggested  that  the  system  should 
include  a  search  pull  down  in  the  metadata 
section  that  allows  for  the  search  of  cargo, 
making  it  easier  to  narrow  down  the  hits. 
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CMA  track  data  was  primarily  limited  to 
tracks  within  the  AOR.  Lack  of  track  data 
outside  the  AOR  limited  the  user’s  ability  to 
analyze  maritime  movement  patterns. 
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CMA:  Gaps  in  track  and  metadata  also 
affected  analysis  of  data  within  a  node  and 
affected  the  ability  to  collaborate  between 
nodes. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

QRA 

75 

Technical 

Performance: 

Information 

Retrieval 

CMA  had  a  number  of  search  capabilities  that 
are  easy  to  use.  There  are  some  deficiencies 
in  advanced  search,  searching  geographic 
areas,  and  partial-match  performance. 

Specifics  are  listed  in  Appendix  E,  System 
Performance. 
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Comprehensive  Maritime  Awareness  (CMA)  - 
-  The  general  consensus  was  CMA  made  it 
easier  to  do  their  job,  but  they  did  not  agree 
that  it  made  it  faster.  "Unless  I  had  an 
extremely  complicated  task  I’m  not  sure  CMA 
would  be  faster  than  consulting  individual 
intelligence  sources,  perhaps  because  I  work 
quickly  with  computers  and  less  complicated 
programs  don’t  have  the  same  technological 
glitches  that  sometimes  thwarted  CMA  (such 
as  losing  the  map  feature  -  the  "halo  of 
death").  (FAIRGAME) 
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The  map  is  probably  the  most  frustrating 
feature  to  use  -  for  example  you  cannot  draw  a 
box  and  simultaneously  use  the  ruler.  Also, 
CMA  runs  significantly  slower  than  some 
Baseline  tools  and  occasionally  the  benefits  of 
the  fused  Intel  is  not  worth  the  wait  (that  is,  it 
didn’t  not  provide  significantly  more 
information  than  looking  on  separate  but  faster 
baseline  tools).  Also  some  servers  contained 
different  information  than  others  and  that  was 
frustrating  and  slowed  me  down." 
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CMA  Alerts:  Was  unable  to  set  up  alerts  and 
tripwires  to  notify  of  transit.  Information  on 
cargo  and  individuals  on  the  vessel  was 
limited. 
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Based  on  survey  responses  (n=19),  CMA 
provides  users  with  data  that  is  relevant  and 
helpful  to  their  analysis  of  conditions  of 
interest  and  CMA  facilitated  the  reliable  and 
timely  analysis  of  maritime  information.  With 
regards  to  reliability,  users  reported  that  it  is 
important  to  be  able  to  view  underlying  data 
provided  by  CMA  in  evaluating  tracks. 
Regarding  timeliness,  users  reported  that 
being  able  to  access  data  from  one  source 
(CMA)  was  faster  than  existing  procedures. 
Users  also  stated  that  it  made  their  job  easier 
to  be  able  to  correlate  and  store  information  in 
a  single  system.  Finally,  63%  of  users  strongly 
agreed  that  response  time  was  adequate. 
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Sixty-three  percent  of  users  "strongly  agreed" 
and  37%  of  users  "agreed"  the  ability  to  view 
the  underlying  data  that  is  used  in  CMA’s 
track  development  is  important  to  them  in 
evaluating  a  track. 

speed 

2 

CMA 

TW08 

230 

Technical 

Performance: 

Information 

Retrieval 

Sixty  percent  of  users  "strongly  agreed"  and 
42%  of  users  "agreed"  that  CMA’s  ability  to 
automatically  identify  differences  in  a  vessel’s 
reported  name,  MMSI,  IMO,  SCONUM,  call 
sign,  emitter  parametrics,  cargo  or  crew  assists 
them  in  their  analysis. 
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The  user’s  ability  to  monitor  vessel,  person, 
and  cargo  data  was  severely  degraded  by  gaps 
in  track  data  coverage.  When  a  node’s  CMA 
server  was  down,  or  data  was  not  transmitted, 
the  data  not  received  was  not  recoverable. 

The  Naval  Research  Laboratory  (NRL)  limited 
the  National  Technical  Means  (NTM)  data 
source  input  to  CMA  to  14  hours  per  day  and 
filtered  the  data  that  was  provided.  This 
resulted  in  a  gap  of  data  which  had  a  negative 
impact  across  all  AORs. 
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The  percentage  of  worldwide  cargo  data  fed  to 
CMA  by  Global  Trader  was  not  known.  The 
percentage  of  worldwide  cargo  data  available 
via  Cargo  Link  was  not  known. 
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The  user  had  no  knowledge  of  foreign-to- 
foreign  cargo  sharing  agreements  or  the 
amount  of  cargo  data  actually  available,  which 
affected  the  ability  to  build  and  evaluate  cargo 
queries. 
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Usability.  The  operator  felt  that  Google  Earth 
was  easy  to  learn,  and  use,  finding  no 
difficulty  in  using  all  of  its  features.  The 
system  was  a  quick  geospatial  reference  for 
developing  awareness  on  a  VOI  as  well  as 
finding  information  on  them. 
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Law  Enforcement  Information  Exchange 
(LInX).  LInX  was  rated  by  all  observers  as 
accessible,  reliable,  and  usable.  The  ease  of 
going  back  and  forth  between  the  Portal  and 
LInX  to  exchange  information  from  one 
system  to  another  was  noted.  LInX  was 
accessible  via  the  FP  Portal.  Participants 
reported  excellent  ability  to  move  between  FP 
Portal  and  LInX.  The  information  available 
through  LInX  was  rated  as  moderate  to  high 
value,  although  one  respondent  wanted  more 
depth  of  social  linkage  information. 

Respondents  generally  trusted  the  sources  that 
provided  information  to  LInX. 
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Other  than  the  collaboration  tools,  LINX  was 
viewed  as  redundant  and  time  consuming. 
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MASTER:  Respondents  agreed  they  could 
track  vessels  as  needed.  MASTER  does  a  very 
good  job  of  pulling  vessel  track  history, 
provides  named  data  that  current  systems  do 
not,  and  facilitates  vessel  tracking  with  its 
graphical  display  and  ability  to  compare 
tracks. 
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One  user  stated  that  MASTER  completes  tasks 
in  20  minutes  that  would  take  a  GALE  Lite 
user  10  hours  because  GALE  Lite  users  have 
to  manually  stitch  tracks. 
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In  MASTER,  they  differed  on  whether  all  the 
data  sources  needed  to  perform  their  tasks 
were  available  in  MASTER.  Sealink  was 
suggested  as  a  data  source  that  could  be 
integrated  with  MASTER. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

MASTER 

TRRLOE 

43 

Technical 

Performance: 

Information 

Retrieval 

The  MASTER  archive  database  does  not 
include  cargo  information;  it  contains  only 
cold  posits  and  associated  data.  Some  users 
used  Cargo  Link  instead  of  MASTER  as  a 
source  of  cargo  information  because  they 
judged  its  database  to  be  more  extensive  than 
master’s  in  this  regard.  Users  indicated 
they  need  the  ability  to  select  individual  cargo 
results  and  draw  the  tracks  on  the  map  in  a 
manner  similar  to  the  track  query  select/draw 
function. 
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MASTER:  MASTER  provides  a  single  source 
of  tools  which  are  superior  to  legacy,  but  users 
would  prefer  the  two  systems  (MASTER  and 
legacy)  to  interact.  Users  typically  identified 
the  correct  tracks  in  their  assigned  scenarios, 
but  various  usability  issues  were  observed 

Some  users  questioned  the  completeness  or 
accuracy  of  the  information  in  MASTER’S 
database.  Several  users  used  SeaLink  to  find 
information  on  a  port  and  to  correlate  old  ship 
names  with  new  ship  names.  One  user  utilized 
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the  PMIC  web  page  in  attempting  to 
determine  the  destination  for  a  particular  ship. 
Two  users  preferred  ASA  over  MASTER 
because  it  is  both  faster  and  easier  to  use,  not 
because  it  is  more  accurate  than  MASTER. 
Respondents  agreed  that  they  could  count  on 
the  pre-defined  alerts  in  MASTER  to  detect 
conditions  of  interest.  Because  alerts  are  not 
easy  to  set  up  using  alert  management  and 
alert  channels,  one  respondent  recommended 
that  alerts  be  based  on  completed  searches  and 
that  an  option  be  provided  to  save  that  search 
as  an  alert. 
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Respondents  differed  on  whether  MASTER 
reduced  their  workload  compared  to  current 
methods.  1 1  of  the  18  responses  to  this  item 
agreed  (4  strongly)  that  MASTER  reduced 
their  workload,  but  the  other  7  responses 
disagreed.  Comments  supporting  MASTER 
noted  its  analytical  capabilities  and  ability  to 
streamline  multiple  activities  into  one  system, 
saving  significant  time.  One  respondent  noted 
that  MASTER  reduced  his  workload  by  25%, 
but  would  be  even  more  effective  if  the  ship’s 
beneficiary  country  was  included  in  the  query 
info.  Dissenting  comments  included: 

MASTER  was  not  easily  accessible;  it  cannot 
locate  vessels  in  real  time  or  find  a  first-time 
vessel  with  no  history;  its  controls  are 
awkward;  and  the  computers  were  slow. 
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Criticisms  noted  that  smaller  vessels  with 
SCONUMs  are  hard  to  detect,  the  archive 
database  is  slow,  some  data  are  not  in  real 
time,  and  a  48-hour  history  is  sometimes 
insufficient. 
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MASTER:  Respondents  agreed  MASTER  is 
very  useful  for  maritime  analysis.  The  track 
analysis  and  plotting  features  are  excellent 
tools,  although  the  large  amount  of 
information  MASTER  provides  may  hamper  a 
rapid  analysis.  Respondents  also  agreed 
master’s  capability  to  integrate  multi¬ 
source  intelligence  and  information  with  fused 
SuperTrack  data  is  valuable.  It  saves  time, 
facilitates  track  fusion  and  analysis,  and 
centralizes  vessel  data. 
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MASTER:  Some  users  questioned  the 
completeness  or  accuracy  of  the  information 
in  master’s  database.  Several  users  used 
SeaLink  to  find  information  on  a  port  and  to 
correlate  old  ship  names  with  new  ship  names. 
One  user  utilized  the  PMIC  web  page  in 
attempting  to  determine  the  destination  for  a 
particular  ship.  Two  users  preferred  ASA 
over  MASTER  because  it  is  both  faster  and 
easier  to  use,  not  because  it  is  more  accurate 
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than  MASTER.  Respondents  differed  on 
whether  MASTER  processes  multiple-source 
intelligence  faster  than  current  methods.  10  of 
the  21  responses  to  this  item  agreed  that  it  did, 
but  the  other  1 1  disagreed.  Criticism  focused 
on  master’s  slowness,  especially  its  slow 
query  responses  and  map  updates.  Current 
tools  (GALE,  Sealink)  were  described  as 
being  much  faster  than  MASTER.  Another 
drawback  is  MASTER’S  current  limit  of  one 
vessel  per  query. 

58 

Technical 

Performance: 

Information 

Retrieval 

Maritime  Integrated  Domain  Awareness 
Solution  (MIDAS).  All  three  operators 
reported  that  MIDAS  enhanced  their  ability  to 
investigate  concerns  regarding  VOIs  or 
persons  of  interest  and  to  understand  why  a 
vessel  is  suspicious  /  threatening.  Two  of  the  3 
said  it  enhanced  their  ability  to  display  a  user- 
defined  picture  of  the  operational 
environment,  and  that  it  provides  access  to 
relevant  commercial  and  law  enforcement 
data.  All  agreed  that  what  is  'relevant  data'  in 
military  operations  is  relative  and  changeable; 

2  of  3  agreed  that  the  MIDAS  threat  picture  is 
able  to  be  set  to  show  relevance  for  a 
particular  military  situation.  Operators 
generally  agreed  that  they  needed  to  have 

MDA  at  both  classified  and  unclassified 
levels;  3  suggested  that  MIDAS  would  be  best 
for  UNCLAS  data,  and  2  others  noted  the 
need  to  integrate  the  unclassified  data  'up'  with 
the  classified  data.  MIDAS  data  was 
mentioned  as  possibly  complementing  GCCS 
data  for  a  fuller  operational  picture. 

operational 
utility  for 

VOIs 

3 

MIDAS 

TWOS 

60 

Technical 

Performance: 

Information 

Retrieval 

MIDAS:  All  respondents  agreed  that  MIDAS 
enables  users  to  make  adjustments  to  detection 
and  assessment  components  to  respond  to 
changes  in  the  military  environment.  Two 
respondents  thought  that  MIDAS  would 
enhance  their  ability  to  perform  the  following 
tasks,  and  one  respondent  was  neutral: 
integrate  and  fuse  relevant  cross-type  data 
from  multiple  sources  and  using  cross-type 
data  to  identify  anomalous  or  threatening 
behavior,  in  time  to  initiate  an  operational 
response.  Two  respondents  thought  that 

MIDAS  would  enhance  their  ability  to  use 
commercial  and  law  enforcement  databases  to 
verify  and  analyze  information  returned  by 
boarding  parties,  and  one  respondent  was 
neutral.  All  three  survey  respondents  thought 
the  MIDAS  would  enhance  their  ability  to 
drill  down  into  data  from  multiple  sources 
regarding  potential  threats. 

fusion  of  data 

3 

MIDAS 

QRA 

135 

Technical 

Performance: 

Information 

PANDA:  Most  users  were  very  satisfied  or 
satisfied  with  the  quality  of  information,  in 
terms  of  its:  availability  (64%),  completeness 

data  & 

processing 

completeness. 

3 

PANDA 

TWOS 

46 
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Retrieval 

(74%),  accuracy  (78%),  timeliness  (71%), 
usefulness  (78%),  and  relevance  (71%) 
(percentages  indicate  percentage  that  were 
satisfied/  very  satisfied). 

correctness, 

conflicts, 

timeliness 

108 

Technical 

Performance: 

Information 

Retrieval 

PANDA:  Most  users  indicated  they  would 
trust  the  information  in  PANDA  to  a  great 
extent  or  moderate  extent  (86%  total).  One 
user  indicated  that  the  information  was  very 
"thorough"  but  other  users  indicated  that  they 
would  trust  the  system  more  once  they 
understood  more  about  the  sources  or  if 
known  reliable  sources  were  added. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

PANDA 

TW08 

28 

Technical 

Performance: 

Information 

Retrieval 

Users  also  noted  that  certain  information  is 
stripped  from  the  Tripwire  database  before  it 
reaches  the  user,  but  what  is  stripped  is 
unknown. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

Tripwire 

TRRLOE 

51 

Technical 

Performance: 

Information 

Sharing 

Spiral- 1  tools  provided  limited  capability  to 
fuse  data  across  AORs  and  security 
classification  domains.  The  user  had  a  limited 
ability  to  view  a  composite  track  with  all 
gathered  information.  The  tools  did  not  allow 
for  user-defined  pedigree  rule  sets. 

fusion  of  data 

2 

All  MDA 

Tech 

TW08 

66 

Technical 

Performance: 

Information 

Sharing 

Spiral- 1  did  not  provide  new  toolsets  to 
enhance  collaboration  between  nodes.  Spiral- 
1  technologies  did  allow  for  the  easy  export  of 
search  response  data  for  use  with  existing 
collaborative  tools.  The  collaboration  tool  of 
choice  was  MS  Chat.  NCIS  used  the  web- 
based  Force  Protection  Portal  for  cross  AOR 
collaboration.  The  portal  acted  as  a  central 
repository  for  all  information  concerning  the 
investigative  scenarios. 

features  for 
collaboration 

2 

All  MDA 

Tech 

VBSS 

125 

Technical 

Performance: 

Information 

Sharing 

Some  users  experienced  situations  when  there 
was  conflicting  information  when  they 
collaborated  with  users  on  other  systems.  For 
example,  one  time  the  name  of  a  vessel  was 
completely  different  from  the  name  listed  in 
all  other  systems,  even  though  other 
identifying  features  of  the  vessel  matched. 

Also  sometimes  it  doesn’t  pull  in  all  the 
features  from  Cargolink/  Sealink  that  it  could, 
which  was  frustrating.  (FAIRGAME) 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

All  MDA 

Tech 

TW08 

281 

Technical 

Performance: 

Information 

Sharing 

The  NPS  assessment  team  is  concerned  that 
effects  of  new  information  flow  at  higher 
volume  may  influence  decision  biases  and 
processes  in  unexpected  ways. 

workload 

2 

All  MDA 

Tech 

Workflow 

PEW 

285 

Technical 

Performance: 

Information 

Sharing 

The  NPS  assessment  team  is  concerned  that 
intelligence  productivity  using  MDA 
technologies  rise  above  operational  capacity. 

workload 

2 

All  MDA 

Tech 

Workflow 

PEW 

76 

Technical 

Performance: 

Information 

Sharing 

There  were  significant  differences  in 
information  available  at  different  nodes. 

data  & 

processing 

completeness, 

correctness, 

conflicts. 

1 

All  MDA 

Tech 

QRA 
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timeliness 

56 

Technical 

Performance: 

Information 

Sharing 

Forty-seven  percent  of  users  "agreed,"  37% 
"strongly  agreed,"  1 1%  "n/a,"  and  5% 
"disagreed"  that  sharing  CMA  data  with  other 
units  and  government  agencies  helped  them  to 
gather  and  share  information  on  vessels  of 
interest. 

sharing  data 

3 

CMA 

TWOS 

54 

Technical 

Performance: 

Information 

Sharing 

MDA  Spiral- 1  was  not  a  composite  system 
and  the  ability  to  aggregate  and  replicate 

MDA  data  on  a  global  scale  was  not  possible. 
CMA  did  not  replicate  its  database  globally; 
each  node  maintained  its  own  local  database. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

n.a. 

67 

Technical 

Performance: 

Information 

Sharing 

Degradation  of  communications  capability 
was  observed  at  one  node  due  to  bandwidth 
limitations  while  conducting  multiple 
exercises.  The  node  was  required  to  timeshare 
bandwidth  to  accommodate  each  event. 
Communications  capability  should  be 
considered  if  there  are  intentions  to  deploy 
these  capabilities  afloat  or  to  a  shore-based 
unit  with  limited  capabilities. 

system 

reliability 

2 

CMA 

QRA 

127 

Technical 

Performance: 

Information 

Sharing 

All  users  surveyed  by  NPS  during 

FAIRGAME  viewed  CMA  as  an  effective 
collaboration  tool,  although  half  had 
experienced  conflicting  information  when  they 
collaborated  with  other  operators  on  a  Vol. 
(FAIRGAME) 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

FAIRGA 

ME 

251 

Technical 

Performance: 

Information 

Sharing 

CMA  -  Users  were  mixed  regarding  whether  it 
was  easy  to  (1)  collaborate  on  a  Vol  and  (2) 
cross  reference  tracks  being  processed  by 
operators  using  other  systems.  Users  were 
mixed  regarding  how  easy  CMA  made  it  for 
them  to  collaborate  about  the  same 
information  with  users  on  other  systems. 
(FAIRGAME)  Yet,  they  all  agreed  it  as  easy 
to  cross  reference  information  on  Vols  with 
operators  using  other  systems. 

features  for 
collaboration 

2 

CMA 

FAIRGA 

ME 

254 

Technical 

Performance: 

Information 

Sharing 

Fingerprints,  iris  images,  and  facial  images 
were  successfully  captured  and  transmitted  for 
analysis. 

transmission 
of  data 

3 

E-MIO 

Wireless 

EC08 

EMIO 

255 

Technical 

Performance: 

Information 

Sharing 

Communications  from  a  ship’s  interior  was 
successful. 

transmission 
of  data 

3 

E-MIO 

Wireless 

EC08 

EMIO 

256 

Technical 

Performance: 

Information 

Sharing 

Detain/don’t  detain  messages  were 
successfully  communicated  to  the  tactical 
units  (communications  were  with  test  systems, 
not  current  operational  systems). 

transmission 
of  data 

3 

E-MIO 

Wireless 

EC08 

EMIO 

55 

Technical 

Performance: 

Information 

Sharing 

MDA  Spiral- 1  was  not  a  composite  system 
and  the  ability  to  aggregate  and  replicate 

MDA  data  on  a  global  scale  was  not  possible. 
CMA  did  not  replicate  its  database  globally, 
each  node  maintained  its  own  local  database. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

Global 

Trader 

QRA 

48 
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232 

Technical 

Performance: 

Information 

Sharing 

Maritime  Domain  Awareness  Data  Sharing 
Community  of  Interest  (MDA  DS  COI).  All 
observers  reported  being  able  to  access  the 
MDA  DS  COI  GMMS  website.  Also,  all 
observers  indicated  the  operator’s  information 
sharing  attempts  were  seamless.  However,  one 
observer  reported  that  although  information 
was  accessible,  not  all  data  were  available  in 
all  geographic  areas.  Eighty  percent  of 
observers  reported  data  was  available  when 
needed  through  MDA  DS  COI  GMMS, 
however,  comments  indicated  the  Google 

Earth  interface  did  not  work  and  that  the 
website  was  slow.  Sixty-six  percent  of  users 
agreed  or  strongly  agreed  that  the  MDS  DS 

COI  provided  a  single  site  for  data  producers 
and  consumers  to  share  unclassified  MDA 
data.  Regarding  what  was  most  liked, 
respondents  mentioned  ease  of  use  and  Google 
maps.  Regarding  what  was  least  liked, 
respondents  mentioned  lack  of  usable  data  — 
services  up  and  down;  errors  received  when 
requesting  data,  and  inability  to  declutter  or 
zoom  on  maps. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

Google 

Earth 

QRA 

68 

Technical 

Performance: 

Information 

Sharing 

Google  Earth  has  no  embedded  collaboration 
tool  included,  thus  it  was  not  possible  to  send 
information  from  Google  Earth  to  other 
systems. 

sharing  data 

1 

Google 

Earth 

TRRLOE 

10 

Technical 

Performance: 

Information 

Sharing 

master’s  strengths  include  its  ability  to 
support  information  sharing  among  users, 
receive  intelligence  from  sources  at  the 

SECRET  level  and  below,  create  watch  areas 
and  briefing  materials,  and  define  the 
operational  picture  to  maintain  awareness  of 
the  maritime  environment. 

misc 

3 

MASTER 

TWOS 

233 

Technical 

Performance: 

Information 

Sharing 

Maritime  Domain  Awareness  Data  Sharing 
Community  of  Interest  (MDA  DS  COI).  All 
observers  reported  being  able  to  access  the 
MDA  DS  COI  GMMS  website.  Also,  all 
observers  indicated  that  the  operator’s 
information  sharing  attempts  were  seamless. 
However,  one  observer  reported  that  although 
information  was  accessible,  not  all  data  were 
available  in  all  geographic  areas.  Eighty 
percent  of  observers  reported  data  was 
available  when  needed  through  MDA  DS  COI 
GMMS,  however,  comments  indicated  that  the 
Google  Earth  interface  did  not  work  and  that 
the  website  was  slow.  Sixty-six  percent  of 
users  agreed  or  strongly  agreed  that  the  MDS 

DS  COI  provided  a  single  site  for  data 
producers  and  consumers  to  share  unclassified 
MDA  data.  Regarding  what  was  most  liked, 
respondents  mentioned  ease  of  use  and  Google 
maps.  Regarding  what  was  least  favorable, 
respondents  mentioned  lack  of  usable  data  — 
services  up  and  down;  errors  received  when 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

MDADS 

COI 

TWOS 

49 
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requesting  data,  and  inability  to  declutter  or 
zoom  on  maps. 

293 

Technical 

Performance: 

Information 

Sharing 

MIDAS:  Most  respondents  rated  the 
following  capability  as  important,  sharing 
unclassified  data  with  other  agencies, 
militaries  and  non-traditional  partners;  making 
decisions  about  asset  allocation  to  address 
potential  threats;  having  a  single  User-Defined 
Operational  Picture  or  access  to  MDA 
information. 

policies  for 
sharing  data 

2 

MIDAS 

TW08 

44 

Technical 

Performance: 

Information 

Sharing 

PANDA:  Regarding  predicting  deviation, 
most  users  (85%)  responded  that  detecting 
deviation  is  a  useful  capability  and  all  users 
responded  that  alerting  for  deviation  is  a 
useful  capability.  Most  users  responded  that 
the  capability  of  classifying  alerts  into 
categories  (86%)  and  that  PANDA’ s 
classification  of  alerts  (86%)  were  useful.  User 
responses  were  mixed  about  whether 
panda’s  explanation  for  the  deviation  was 
useful.  Information  that  users  would 
communicate  to  others  about  deviations 
included  %  likelihood  a  vessel  would  deviate, 
context  information  such  as  ship 
parts/ spares/repair,  and  crew  change/illness 
(currently  not  available  in  PANDA),  other 
vessels  in  the  area,  whether  a  vessel  maybe  a 
potential  threat  or  information  for  a  task. 

sharing  data 

3 

PANDA 

TW08 

70 

Technical 

Performance: 

Interoperabili 

ty 

Some  CMA  client  workstations  using  legacy 
Internet  Explorer  6  could  not  operate  CMA. 
CMA  is  designed  to  operate  on  Internet 

Explorer  7.  The  workaround  was  to  use  the 
Firefox  web  browser. 

interoperation 
with  other 
systems 

2 

CMA 

TW08 

71 

Technical 

Performance: 

Interoperabili 

ty 

CMA  requires  JAVA  1.6.  Other  host  systems 
use  legacy  version  JAVA  1.4.  When  CMA  is 
loaded  on  the  same  hardware  as  GCCS-M,  the 
JAVA  1.6  causes  the  local  GCCS  workstation 
to  crash.  The  workaround  is  that  CMA  must 
be  loaded  on  separate  hardware. 

interoperation 
with  other 
systems 

2 

CMA 

QRA 

74 

Technical 

Performance: 

Interoperabili 

ty 

Each  node  maintains  its  own  CMA  database 
with  no  two  nodes  having  the  same 
information. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

QRA 

73 

Technical 

Performance: 

Interoperabili 

ty 

User  feedback  indicated  FASTC2AP  on 
SIPRNET  would  be  more  useful/capable  with 
additional  GENSER  track  history  feeds  not 
available  on  CENTRIX. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

FASTC2AP 

TRRLOE 

72 

Technical 

Performance: 

Interoperabili 

ty 

When  using  CMA  and  Google  Earth  4.2 
simultaneously,  system  crashes  resulted. 

Google  Earth  4.2  had  to  be  configured  to  use 
DIRECTX  graphics  setting  instead  of 

OpenGL  graphics  setting  because  the  version 
of  OpenGL  installed  on  the  CMA  workstation 

interoperation 
with  other 
systems 

2 

Google 

Earth 

QRA 
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causes  Google  Earth  to  crash. 

140 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Because  MDA  Spiral- 1  was  not  a  composite 
system,  users  were  required  to  switch  between 
windows  and  classification  domains  multiple 
times,  causing  them,  at  times,  to  become 
confused  as  to  which  domain  they  were 
looking  at.  The  confusion  caused  users  to  lose 
track  of  the  original  security  classification  of 
the  information  they  wished  to  disseminate. 

integration  of 

multiple 

displays 

2 

All  MDA 

Tech 

QRA 

114 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Usability.  Both  CMA  operators  thought  the 
technology  was  easy  to  learn  and  use.  Both 
CMA  users  strongly  agreed  it  was  easy  to 
develop  and  maintain  situation  awareness  on  a 
vessel  of  interest  (VOI), 

usability 

3 

CMA 

QRA 

130 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Users  were  able  to  easily  create  watch  areas 
and  do  what  they  need  to  do  for  a  mission. 
However,  it  should  be  noted  that  many  users 
reported  "N/A"  with  regard  to  ease  of  being 
able  to  create  briefing  materials  and  methods 
for  creating  watch  areas. 

usability  of 
briefs 

3 

CMA 

QRA 

146 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Most  users  agreed,  or  strongly  agreed,  CMA 
was  easy  to  learn  and  use,  as  evidenced  by  the 
following:  "With  the  exception  of 
technological  glitches  that  occasionally 
hindered  CMA’s  performance  I  was  able  to 
learn  and  maneuver  the  program  very  quickly 
-  it  probably  took  less  than  two  hours  of 
training  and  a  few  examples  to  understand  all 
the  major  features."  Users  were  mixed  about 
the  helpfulness  of  the  online  help  and  about 
whether  it  was  easy  to  use  all  the  features 
included  in  CMA.  (FAIRGAME) 

usability  of 
map 

3 

CMA 

EC  EMIO 

155 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

CMA  features  they  liked:  (1)  Nice  search 
function,  if  a  little  slow;  (2)  Strongly  agreed 
was  able  to  process  and  maintain  situation 
awareness  on  more  than  one  Vol  at  a  time 
more  easily  than  with  my  previous  system: 
"Especially  with  the  "Map  Viewer"  and  ‘View 
Details"  as  separate  windows  - 1  love  the 
tabbed  browsing." 

usability  of 
search 

3 

CMA 

VBSS 

161 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Watch  areas:  Very  intuitive  - 1  like  that  it 
saves  them  and  they  are  easy  to  search  in 

usability  of 
search 

3 

CMA 

TWOS 

162 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Bounding  Box:  Was  able  to  mark  and  track 
areas  w/  precision. 

usability  of 
map 

3 

CMA 

VBSS 

164 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Name  variants:  Was  able  to  limit  searching.  In 
addition  was  able  to  find  specific  vessels  if  I 
had  specific  information. 

usability  of 
search 

3 

CMA 

VBSS 

166 

Warfighter 

Acceptance: 

Tracking  data:  Was  able  to  see  the  exact  posits 
in  the  past  and  corroborate  with  the  reporting 

fusion  of  data 

3 

CMA 

QRA 
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Human- 

System 

Interaction 

168 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Associate  data:  This  was  the  most  helpful  in 
that  in  addition  to  the  ship  information 
provided:  Company  name,  location,  master, 
crew  and  other  information  associated  with  the 
vessel.  Again  as  a  ct  analyst  want  to  look  at 
more  than  just  the  vessel  and  where  it  is 
headed,  want  as  much  information  as  possible 
that  associates  the  vessel  with  other  entities. 

fusion  of  data 

3 

CMA 

VBSS 

183 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

CMA:  It  was  simply  a  matter  of  click  on  the 
vessel  and  most  of  the  relevant  information 
was  placed  before  me.  If  I  were  a  ship  tracker 
this  is  great.  In  addition  I  would  be  able  to  use 
the  additional  information  for  the  CT  mission. 

I  am  not  so  much  interested  in  the  path  of 
vessels  but  rather  who/  what  is  on  the  vessel." 
(FAIRGAME) 

fusion  of  data 

3 

CMA 

VBSS 

189 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

How  the  system  applied  to,  or  could  be  used 
for,  the  jobs  with  which  they  were  familiar 
was  clear. 

operational 

utility 

3 

CMA 

TRRLOE 

192 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Most  users  agreed,  or  strongly  agreed,  CMA 
was  easy  to  use,  and  all  agreed  they  were  able 
to  access  and  use  the  online  help. 

(FAIRGAME) 

usability  of 
help 

3 

CMA 

VBSS 

240 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

It  was  easy  for  CMA  users  to  associate  tipper/ 
Intel  information  with  the  correct  VOL  CMA 
showed  recent  alerts  on  the  vessel  and  made  it 
easy  to  see  the  tippers. 

fusion  of  data 

3 

CMA 

TRRLOE 

35 

Warfighter 

Acceptance: 

Human- 

System 

Interaction 

Suggestions  for  Improvement  and  Useful 
Features.  Both  CMA  users  experienced 
difficulties  using  the  alerting  system  as  well  as 
the  hyper  graph.  Participants  noted  that  when 
searching  on  CMA  using  a  partial  search,  the 
system  did  not  always  pull  up  all  tracks 
containing  a  portion  of  the  name.  One  user 
recommended  adding  a  feature  to  provide  the 
capability  to  use  range  and  bearing  from  a 
point  in  a  circle  to  form  a  search  area.  This 
user  also  suggested  including  the  ability  to  use 
a  line  or  border  to  search  for  any  track 
crossing  a  line  rather  than  a  box  as  is  done  in 
the  current  system. 

usability  of 
map  and 
hypergraph 
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CMA  users  suggested  that  the  system  should 
include  a  search  pull  down  in  the  metadata 
section  that  allows  for  the  search  of  cargo, 
making  it  easier  to  narrow  down  the  hits. 

usability  of 
menu 
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CMA  advance  searches  were  not  intuitive. 
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142 

Warfighter 
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Search  capabilities  in  CMA  and  FASTC2AP 
are  easy  to  use  but  there  are  a  number  of 
improvements  that  would  improve  their 
usability  for  operators  and  expand  types  of 
searches  that  can  be  undertaken.  An  appendix 
lists  a  number  of  improvements  that  operators 
have  requested,  listed  under  Human-System 
Interaction.  Many  suggestions  have  to  do  with 
constructing  search  areas  and  search 
parameters. 

usability  of 
search 
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The  map  is  probably  the  most  frustrating 
feature  to  use  -  for  example  you  cannot  draw  a 
box  and  simultaneously  use  the  ruler.  Also, 
CMA  runs  significantly  slower  than  some 
Baseline  tools  and  occasionally  the  benefits  of 
the  fused  Intel  is  not  worth  the  wait  (that  is,  it 
didn’t  not  provide  significantly  more 
information  than  looking  on  separate  but  faster 
baseline  tools).  Also  some  servers  contained 
different  information  than  others  and  that  was 
frustrating  and  slowed  me  down." 

fusion  of  data 
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Acceptance: 
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To  an  extent  it  is  helpful  so  see  a  fused  picture 
of  all  the  Intel  related  to  a  specific  vessel,  but 
ONLY  when  this  was  available  (sometimes 
there  just  isn’t  much  more  available  on  a 
vessel  than  what  is  in  Sealink/SeaWatch)  or 
when  complicated  scenarios  demanded  the  use 
of  CMA  instead  of  a  less  advanced  tracking 
system."  (FAIRGAME) 

usability 
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Three  of  four  users  surveyed  at  FAIRGAME 
indicated  there  were  feature  they  wanted  to 
perform  with  the  system  that  they  did  not 
understand.  For  example:  "During  the  exercise 
there  was  a  lot  of  confusion  about  being  able 
to  track  vessels  that  come  near  a  specific 
vessel  at  any  time  during  its  track  - 1  still  don’t 
think  this  function  is  truly  possible." 

leamabilty 
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CMA 
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CMA:  There  were  also  actions  they  wanted  to 
perform  with  the  system  that  were  not 
available,  e.g.,  "I  think  the  map  could  be  more 
user-friendly  -  as  I  mentioned  earlier  one 
should  be  able  to  use  the  bounding  box  and 
then  the  ruler  without  using  the  box.  Also,  it 
would  be  helpful  if  there  were  a  way  to  merge 
tracks  that  the  system  lists  as  separate  hits  due 
to  slight  discrepancies  but  are  clearly  the  same 
vessel. 

usability  of 
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2 

CMA 
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Other  comments  regarding  human-system 
interaction  for  CMA:  (1)  Advanced  search  is 
tricky  and  not  very  intuitive  but  does  the  job; 

(2)  the  map  could  be  more  user-friendly  -  one 
should  be  able  to  use  the  bounding  box  and 
then  the  ruler  without  using  the  box;  (3)  it 
would  be  helpful  if  there  was  a  way  to  merge 
tracks  that  the  system  lists  as  separate  hits  due 
to  slight  discrepancies  but  are  clearly  the  same 
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vessel.  (FAIRGAME) 
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View  Details:  Color-coded,  fused  Intel  was 
very  helpful  WHEN  it  was  available.  I  would 
use  CMA  maybe  after  consulting  one  or  two 
baseline  programs  first,  otherwise  it’s 
sometimes  just  too  impractical  to  poke  around 
CMA  when  it  is  running  slowly  or  having 
errors.  However,  if  I  have  a  specific  set  of 
variables  for  a  VOI  (i.e.  past  locations,  people 
on  board,  flag,  etc.)  it  is  a  very  helpful 
resource. 

usability  of 
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CMA  Map:  See  above  comments:  also  there 
are  still  some  kinks  that  need  to  be  worked  out 
(possibly  with  Java)  that  made  the  program 
crash  and  I  had  to  clear  my  Java  files  and 
restart  the  program  -  this  happened  much  too 
often. 

reliability 
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CMA  Alerts:  Was  unable  to  set  up  alerts  and 
trip  wires  to  notify  of  transit.  Information  on 
cargo  and  individuals  on  the  vessel  was 
limited. 

usability  of 
alerts 
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CMA  -  All  four  users,  surveyed  during 
FAIRGAME,  agreed/  strongly  agreed  it  was 
easy  to  develop  and  maintain  situation 
awareness  on  a  Vol.  Users  were  mixed 
(ranged  from  neutral  to  strongly  agree) 
regarding  whether  it  was  easy  to  (1)  find 
information  on  a  Vol;  (2)  associate  tipper/ 
intel  information  with  the  correct  Vol;  and  (3) 
conduct  analysis  on  a  Vol.  (FAIRGAME) 

search 
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Tactical  EMIO  System  (TES)  -  Observations 
of  the  Maritime  BGAN  EMIO  Terminal 
(MEET)  demonstrated  that  (1)  its  design  was 
very  rugged  and  seemed  to  prevent  water 
intrusion;  and  (2)  the  MEET  device  was  not 
light  enough  to  be  carried  on  a  VESS  person. 
The  student  user’s  felt  that  (1)  the  equipment 
should  integrate  the  camera  along  with  the 
biometric  capability;  (2)  the  Tactical  EMIO 
Device  (TED)  had  too  many  functions  to 
organize  the  data;  (3)  they  would  prefer 
required  less  steps  to  configure  the  system;  (4) 
the  EMIO  Gear  was  very  mobile;  and  (5)  the 
graphic  user  interface  was  intuitive  and  simple 
to  use.  (VBSS  School) 
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Accordingly,  the  BFC  manually  entered  into 
the  CoT  messages  the  position  reported  to 
them  over  VCl  chat. 

integration  of 
multiple  tools 
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Usability.  The  operator  felt  that  FASTC2AP 
was  difficult  to  learn  resulting  in  the  user 
having  to  try  the  agents  multiple  times  in  order 
to  understand  what  each  was  providing  in  the 
way  of  results.  The  operator  stated  it  was 
fairly  easy  to  create  agents;  however 
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understanding  the  results  is  difficult.  It  was 
easy  to  compose  agents  and  to  build  alerts  in 
order  to  fill  RFIs,  although  the  system  could 
not  provide  the  details  required  for  some  of 
the  RFIs. 

78 

Warfighter 
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Human- 
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A  FASTC2AP  geospatial  option  would  allow 
greater  situational  awareness  and  agent 
building. 

usability  of 
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Search  capabilities  in  CMA  and  FASTC2AP 
are  easy  to  use  but  there  are  a  number  of 
improvements  that  would  improve  their 
usability  for  operators  and  expand  types  of 
searches  that  can  be  undertaken.  An  appendix 
lists  a  number  of  improvements  that  operators 
have  requested,  listed  under  Human-System 
Interaction.  Many  suggestions  have  to  do  with 
constructing  search  areas  and  search 
parameters. 
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FASTC2AP  has  predefined  alert  agents  and 
the  facility  for  operators  to  define  their  own 
alerts.  The  agents  were  useful  but 
understanding  some  of  the  agent  results  was 
difficult.  This  was  due  to  agent  templates 
being  difficult  to  understand.  Setting  up  new 
agents  is  difficult.  The  system  is  non- 
intuitive. 

usability  of 
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Usability.  The  operator  felt  that  Google  Earth 
was  easy  to  learn,  and  use,  finding  no 
difficulty  in  using  all  of  its  features.  The 
system  was  a  quick  geospatial  reference  for 
developing  awareness  on  a  VOI  as  well  as 
finding  information  on  them. 
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Conducting  analysis  on  a  VOI  with  Google 
Earth  was  easy,  specifically  for  acquiring  their 
last  position;  however  the  VOLPE  AIS  data 
was  limited,  and  not  useful  for  in-depth 
analysis  or  for  providing  tracking  data. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 
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master’s  user  interface  was  judged  to  be 
clear,  consistent,  and  easy  to  understand,  and 
most  respondents  found  it  easy  to  get 

MASTER  to  do  what  they  wanted,  given 
adequate  training  and  practice,  which  are 
needed  to  reach  proficiency.  Usability 
drawbacks  include  multiple  click  controls  (2 
or  3  clicks)  for  various  features  and  slow 
response  times,  especially  to  database  queries. 
master’s  display  also  has  a  tendency  to 
become  too  cluttered  for  effective  viewing 
under  higher  traffic  conditions. 

usability  re: 
clutter  & 
efficiency 
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Its  user  interface  could  be  improved  by 
making  the  vessel  of  interest  category  more 
salient  and  improving  the  mapping  program. 
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MASTER:  Respondents  agreed  that  MASTER 
is  easy  to  use  for  creating  tracks  for  vessels  of 
interest,  although  one  respondent  noted  that 
vessel  tracks  are  archived  too  quickly  to  use 
for  tracking  VOIs.  Recommended  user 
interface  improvements  included:  make  the 
vessel  of  interest  category  more  salient  in 
current  vessel  details,  improve  the  mapping 
program,  and  add  the  ability  to  take  tracks 
from  MASTER  and  transfer  them  into  GALE 
to  improve  EIE  tracking.  One  respondent 
wanted  to  be  able  to  export  track  data  to  Excel 
to  better  compare  tracks  and  to  compile  data 
more  effectively. 

usability 
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Trip  wire,/ Alert:  Simply  unable  to  use.  With 
more  training  am  sure  that  too  could  be 
mastered 
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Maritime  Domain  Awareness  Data  Sharing 
Community  of  Interest  (MDA  DS  COI). 
Feedback  indicates  that  usability  was 
relatively  low.  Users  were  able  to  identify 
suspect  VOIs,  however,  it  was  difficult.  One 
observer  stated  "This  user  interface  needs 
major  improvement  with  respect  to  correlating 
data  relationships  between  popup  windows 
and  the  map,  and  the  track  data  playback 
function.  The  process  requires  the  operator  to 
bounce  between  track  data  popup  spreadsheets 
and  layer  tools  lists.  The  operators  had  to 
manually  document  record  track  identifier 
numbers  to  compare  data  list  on  other 
windows.  This  process  is  prone  to  errors  as 
demonstrated  during  this  event." 
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Predictive  Analysis  for  Naval  Deployment 
Activity  (PANDA):  Most  users  indicated  that 
they  information  provided  in  PANDA  was 
easy  to  understand.  Eighty-six  percent  agreed 
(others  were  neutral)  it  was  easy  to  understand 
the  vessel  list  details;  93%  agreed  (1  user 
disagreed)  the  use  of  color  associated  with 
deviation  types  made  sense;  all  users  agreed  it 
was  easy  to  understand  the  meaning  of  the 
color  coding;  86%  agreed  (others  disagreed) 
that  it  was  easy  to  understand  the  meaning  of 
the  labels  used  for  deviation;  75%  agreed  (1 
was  neutral)  PANDA  made  it  apparent  that  a 
vessel’s  behavior  was  similar  to  previous 
behavior.  Regarding  normalcy:  All  users 
indicated  that  the  PANDA  was  effective  at 
representing  the  normal  behaviors  of  vessels 
in  the  PANDA  survey. 
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The  post-installation  training  was  generic  in 
nature  and  not  tailored  to  the  individual  site 
requirements/mission.  User  feedback 
indicated  that  the  FAIRGAME  scenario 
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construct  provided  a  more  realistic  training 
environment  because  the  scenarios  allowed  for 
more  real-world  applications  of  the 
technologies.  An  official  Navy  training 
pipeline  is  not  in  place  for  MDA  Spiral- 1. 
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Operators  were  able  to  achieve  reasonable 
competence  on  the  systems  after  3  hours  of 
training  and  2  hours  of  use  on  designed 
tactical  problems.  It  is  anticipated  that  a  one 
week  school  on  the  full  Spiral- 1  suite  would 
produce  fully  competent  operators. 
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CINCPACFLEET  is  concerned  about  the 
feasibility  of  learning,  using,  and  maintaining 
new  MDA  technologies  given  that  it  has  a 
small  Intel  unit. 
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ONI  expresses  concern  about  lack  of  lack  of 
re-engineering  of  processes  and  training. 

process  re¬ 
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NAVCENT  is  concerned  about  training  MDA 
technologies. 
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PACFLT  and  NAVCENT  are  concerned  that 
COCOM-specific,  mission-specific  training  be 
provided  with  MDA  systems 
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Most  users  agreed,  or  strongly  agreed,  CMA 
was  easy  to  learn  and  use,  as  evidenced  by  the 
following:  "With  the  exception  of 
technological  glitches  that  occasionally 
hindered  CMA’s  performance  I  was  able  to 
learn  and  maneuver  the  program  very  quickly 
-  it  probably  took  less  than  two  hours  of 
training  and  a  few  examples  to  understand  all 
the  major  features."  Users  were  mixed  about 
the  helpfulness  of  the  online  help  and  about 
whether  it  was  easy  to  use  all  the  features 
included  in  CMA.  (FAIRGAME) 
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CMA  -  Operator  training  on  the  operations 
they  were  asked  to  perform  with  the  systems 
was  minimal.  Within  this  context,  the 
following  observations  can  be  made: 
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They  developed  facility  with  the  systems  in  a 
short  period  of  time,  usually  after  about  Vi 
hour  of  working  with  functionality. 

training  speed 
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It  is  clear  that  two  or  three  days  of  focused 
training  on  the  Spiral- 1  suite  will  result  in 
fully  competent  operators. 
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There  were  instances  where  an  operator 
searched  for  a  function  or  capability  CMA  did 
not  have.  This  could  be  a  training  issue  and/or 
represent  functionalities  that  should  be  added. 

training 
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System 

Training 

system  in  CMA  was  easy  to  understand.  ("I 
might  not  have  been  trained  well  enough  on 
using  it  but  I  did  not  find  the  alerting  system 
especially  helpful,  save  for  the  ability  to  save 
watch  areas  which  is  an  easy  system.") 
Regarding  whether  the  alerting  system  in 

CMA  was  useful  and  relevant,  users  were 
either  neutral  or  disagreed,  e.g.,  "To  set  up  the 
trips  was  not  easy.  Establishing  the  bounding 
boxes  and  setting  up  the  search  parameters 
was  easy  but  needed  further  instruction  on 
alert  set  up."  (FAIRGAME) 
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CMA  -  The  envisioned  CONOPS  for  future 
MDA  will  require  training  and  familiarization 
with  the  new  procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly  collaborate 
with  others,  with  the  exception  of  perhaps 
shared  watch  areas;"  and  (2)  "During  the 
exercise  there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come  near  a 
specific  vessel  at  any  time  during  its  track  - 1 
still  don’t  think  this  function  is  truly  possible." 

training 
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CMA  -  The  envisioned  TTP/SOP  for  future 
MDA  will  require  training  and  familiarization 
with  the  new  procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly  collaborate 
with  others,  with  the  exception  of  perhaps 
shared  watch  areas;"  and  (2)  "During  the 
exercise  there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come  near  a 
specific  vessel  at  any  time  during  its  track  - 1 
still  don’t  think  this  function  is  truly  possible." 
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The  Biometrics  CM  Jump  Kit  with  MOBS 

VI. 3. 3  proved  intuitive.  The  operator  had  15 
minutes  of  training.  His  time  capturing  a  full 
enrollment  decreased  with  each  enrollment. 

By  his  third  enrollment,  he  required  no 
assistance. 
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The  biometric  Jump  Kit  required  only  a  small 
amount  of  training  to  be  used  successfully. 
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Tactical  EMIO  System  (TES)  -  Training  for 
use  of  the  Tactical  EMIO  System  (TES)  was 
conducted  at  the  VBSS  School  in  San  Diego 
and  testing  was  planned  for  two  sessions  at  the 
conclusion  of  two  courses  (June  and  July).  As 
is  so  often  the  case,  the  10  student  system 
users  felt  they  did  not  receive  enough  training, 
and  in  particular,  time  to  practice  operating  the 
system. 
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Observing  the  operation  of  the  Tactical  EMIO 
Device  (TED)  in  the  field  indicated  the 
students  would  require  additional  training  to 
fully  operate  the  TED.  However,  the  students 
were  able  to  use  the  features  to  capture 
required  data.  Students  asked  many  questions 

training 

completeness 

2 

E-MIO 

Wireless 

Workflow, 

PEW 

58 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

during  the  training  which  provided  useful 
feedback  to  learn  the  perspective  of  the  user. 
(VBSS  School) 

277 

Warfighter 

Acceptance: 

System 

Training 

Tactical  EMIO  System  (TES)  -  Students  were 
not  granted  enough  time  to  practice  operating 
all  three  systems  (TED,  TEMP,  and  MEET). 
Observing  the  operation  of  the  Tactical  EMIO 
Device  (TED)  in  the  field  indicated  the 
students  would  require  additional  training  to 
fully  operate  the  TED. 

training  speed 

2 

E-MIO 

Wireless 

VBSS 

100 

Warfighter 

Acceptance: 

System 

Training 

Criticisms  noted  that  smaller  vessels  with 
SCONUMS  are  hard  to  detect,  the  archive 
database  is  slow,  some  data  are  not  in  real 
time,  a  48-hour  history  is  sometimes 
insufficient. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

MASTER 

TW08 

102 

Warfighter 

Acceptance: 

System 

Training 

there  is  a  need  for  more  training  on  Master 

training 

2 

MASTER 

QRA 

205 

Warfighter 

Acceptance: 

System 

Training 

The  MDA  DS  COI  may  provide  a  basic 
anomalous  detection  capability  but  since  the 
users  were  not  trained  on  the  full  capability  it 
was  not  observed. 

training 

completeness 

2 

MDADS 

COI 

TW08 

89 

Warfighter 

Acceptance: 

System 

Usage 

Users  were  unaware  of  technology  manuals 
and  the  Online  Learning  Development 
(OLLD).  The  draft  online  learning  compact 
disc  was  made  available  to  the  nodes  no 
earlier  than  two  days  before  the  start  of 
FAIRGAME.  The  draft  version  of  the  OLLD 
is  sufficient  to  provide  the  user  with  a  baseline 
buttonology  level  of  knowledge.  The 
finalized  version  of  the  OLLD  is  expected  to 
be  made  available  via  web  services  sometime 
in  August  2008. 

training  in 
buttononlogy 

3 

All  MDA 

Tech 

QRA 

88 

Warfighter 

Acceptance: 

System 

Usage 

Beyond  the  tasking  provided  by  the  White 

Cell,  the  users  did  not  know  why  or  when  to 
use  a  specific  technology  to  produce  an 
expected  results.  Users  required  prompting  by 
the  White  Cell,  and  observers,  to  think  outside 
of  the  box  when  looking  for  specific 
information. 

SOP  and 

TTPs 

2 

All  MDA 

Tech 

QRA 

181 

Warfighter 

Acceptance: 

System 

Usage 

During  FAIRGAME,  CMA  was  often  the 
technology  of  choice  when  starting  a  search. 

operational 

utility 

3 

CMA 

QRA 

182 

Warfighter 

Acceptance: 

System 

Usage 

The  Tripwire  system  was  the  technology  of 
choice  for  users  who  were  experienced  with 
the  capabilities  it  provides. 

operational 

utility 

3 

Tripwire 

Workflow, 

PEW 

2 

Warfighter 

Acceptance: 

System 

Utility 

The  majority  of  users  reported  liking  the 
ability  that  CMA  gives  them  to  leave  an  alert 
in  a  vessel  file  that  can  be  used  by  other 
analysts.  Users  also  reported  being  able  to  rely 
on  pre-defined  alerts  to  stay  informed  about 
vessels  of  interest.  Regarding  the  timely 
attribute,  users  reported  that  query  functions 

search 

3 

CMA 

EC  EMIO 

59 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

saved  significant  time.  Specifically,  it  was 
helpful  to  be  able  to  filter  by  location,  time, 
vessel  name  and  vessel  attributes. 

6 

Warfighter 

Acceptance: 

System 

Utility 

CMA  is  viewed  as  enhancing  SA  rapidly. 

Ninety  percent  of  users  either  agreed  or 
strongly  agreed  that  they  could  maintain 
awareness  of  maritime  activity  through  CMA. 
Forty-two  percent  of  users  "agreed,"  26% 
"strongly  agreed,"  16%  "disagreed,"  and  16% 
indicated  "N/A"  that  CMA’s  capability  to 
automatically  acquire  data  on  vessels  of 
interest  from  multiple  databases  is  faster  than 
current  data  acquisition  processes/ systems. 
Forty-seven  percent  of  users  "strongly 
agreed,"  32%  "agreed,"  16%  were  "n/a"  and 

5%  of  users  "disagreed"  that  CMA’s  query 
capabilities  (search  and  advanced  search)  save 
them  considerable  time  in  gathering 
information  on  a  vessel  of  interest  compared 
to  current  system  capabilities.  Fifty-three 
percent  of  users  "strongly  agreed"  and  42% 
"agreed"  that  CMA’s  advanced  search 
capability  to  filter  information  by  location, 
time,  vessel  name,  vessel  attributes,  etc.,  saves 
them  time  compared  with  having  to  do 
multiple  individual  searches. 

search 

3 

CMA 

TWOS 

30 

Warfighter 

Acceptance: 

System 

Utility 

Very  little  time  was  required  for  operators  to 
access  information  from  CMA,  typically  1-10 
min.  This  was  with  pre-selected  ships  where 
information  was  known  to  exist. 

speed 

3 

CMA 

QRA 

38 

Warfighter 

Acceptance: 

System 

Utility 

Participants  thought  conducting  analysis  on  a 
VOI  was  easier  and  much  faster  when  using 
CMA  than  with  their  previous  system. 

speed 

3 

CMA 

TWOS 

61 

Warfighter 

Acceptance: 

System 

Utility 

Forty-two  percent  of  users  "agreed,"  32% 
"strongly  agreed,"  and  26%  indicated  "N/A" 
that  their  ability  to  individually  identify  and 
save  analytic  areas  of  interest  in  CMA  helps 
them  in  their  analytic  work. 

speed 

3 

CMA 

TWOS 

110 

Warfighter 

Acceptance: 

System 

Utility 

Although  the  data  reliability  issue  cited  by 
users  remains  a  concern,  CMA  is  still  a 
capable  system  and  greatly  enhances  the 
warfighter’s  ability  to  maintain  situational 
awareness. 

operational 
utility  for  SA 

3 

CMA 

QRA 

129 

Warfighter 

Acceptance: 

System 

Utility 

Ninety  percent  of  users  either  agreed  or 
strongly  agreed  that  they  could  maintain 
awareness  of  maritime  activity  through  CMA. 

operational 
utility  for  SA 

3 

CMA 

Workflow, 

PEW 

32 

Warfighter 

Acceptance: 

System 

Utility 

During  FAIRGAME,  users  were  mixed 
(responses  ranged  from  disagree  to  agree) 
regarding  whether  they  were  able  to  develop 
awareness  of  VOIs  faster  with  CMA  "Again, 
definitely  when  there  was  a  good  fused  picture 
or  a  complicated  track  I  needed  to  follow.  I 
imagine  for  most  VOIs  it  won’t  be 
significantly  faster,  though."  All  users 
agreed/strongly  agreed  it  was  easy  to  find 

speed 

2 

CMA 

VBSS 

60 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

information  on  a  Vol.  (FAIRGAME) 

124 

Warfighter 

Acceptance: 

System 

Utility 

Comprehensive  Maritime  Awareness  (CMA)  - 
-  The  general  consensus  was  CMA  made  it 
easier  to  do  their  job,  but  they  did  not  agree 
that  it  made  it  faster.  "Unless  I  had  an 
extremely  complicated  task  I’m  not  sure  CMA 
would  be  faster  than  consulting  individual 
intelligence  sources,  perhaps  because  I  work 
quickly  with  computers  and  less  complicated 
programs  don’t  have  the  same  technological 
glitches  that  sometimes  thwarted  CMA  (such 
as  losing  the  map  feature  -  the  "halo  of 
death").  (FAIRGAME) 

speed 

2 

CMA 

VBSS 

79 

Warfighter 

Acceptance: 

System 

Utility 

Google  Earth  fusion  services  provided  some 
utility  but  was  viewed  as  redundant. 

technology 

redundancy 

2 

Google 

Earth 

QRA 

139 

Warfighter 

Acceptance: 

System 

Utility 

Law  Enforcement  Information  Exchange 
(LInX).  LInX  was  rated  by  all  observers  as 
accessible,  reliable,  and  usable.  The  ease  of 
going  back  and  forth  between  the  Portal  and 
LInX  to  exchange  information  from  one 
system  to  another  was  noted.  LInX  was 
accessible  via  the  FP  Portal.  Participants 
reported  excellent  ability  to  move  between  FP 
Portal  and  LInX.  The  information  available 
through  LInX  was  rated  as  moderate  to  high 
value,  although  one  respondent  wanted  more 
depth  of  social  linkage  information. 

Respondents  generally  trusted  the  sources  that 
provided  information  to  LInX. 

usability 

3 

LiNX 

TW08 

81 

Warfighter 

Acceptance: 

System 

Utility 

Other  than  the  collaboration  tools,  LINX  was 
viewed  as  redundant  and  time  consuming. 

technology 

redundancy 

2 

LiNX 

TRRLOE 

294 

Warfighter 

Acceptance: 

System 

Utility 

Information  from  a  stand-alone  MAGNET 
system  was  not  available  due  to  sharing 
agreements  concerning  U.S.  person’s 
information. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

MAGNET 

n.a. 

8 

Warfighter 

Acceptance: 

System 

Utility 

MASTER:  Respondents  agreed  that  they 
could  maintain  awareness  of  the  maritime 
environment  through  the  operational  picture 

operational 
utility  for  SA 

3 

MASTER 

TW08 

104 

Warfighter 

Acceptance: 

System 

Utility 

MASTER:  Respondents  agreed  MASTER  is 
very  useful  for  maritime  analysis.  The  track 
analysis  and  plotting  features  are  excellent 
tools,  although  the  large  amount  of 
information  MASTER  provides  may  hamper  a 
rapid  analysis.  Respondents  also  agreed 
master’s  capability  to  integrate  multi¬ 
source  intelligence  and  information  with  fused 
SuperTrack  data  is  valuable.  It  saves  time, 
facilitates  track  fusion  and  analysis,  and 
centralizes  vessel  data. 

fusion  of  data 

2 

MASTER 

TW08 

82 

Warfighter 

FAIRGAME  user  feedback  indicated  MDA 

technology 

2 

MDADS 

QRA 

61 


Item 

ID 
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Item 

Comments 
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Technology 

Venue 

Acceptance: 

System 

Utility 

DS  COI  has  limited  utility  that  might  be 
accessed  more  as  an  afterthought  or  when  all 
other  systems  had  been  exhausted. 

redundancy 

COI 

217 

Warfighter 

Acceptance: 

System 

Utility 

PANDA:  All  12  users  indicated  PANDA 
would  have  a  positive  effect  on  their  situation 
awareness.  Users  pointed  to  a  variety  of 
PANDA  features  that  would  support  improved 
SA,  including  an  increase  in  understanding  of 
normalcy  and  anomalous  vessel  behavior, 
alerting  to  behavior  that  would  otherwise  be 
missed,  prediction  related  to  ships  inbound  to 

US  ports,  understanding  of  vessel  history,  and 
potential  support  for  "proactive"  analysis. 

operational 
utility  for  SA 

3 

PANDA 

VBSS 

112 

Warfighter 

Acceptance: 

System 

Utility 

Tripwire  Contextual  information  on  vessels  of 
interest  and  persons  of  interest  is  outstanding. 

operational 
utility  for 

VOIs 

3 

Tripwire 

VBSS 

62 


6.1  Operations  Performance  Results 


The  assessment  area  “operations  performanee”  concerns  capabilities  required  to  carry  out 
specific  operations:  Vessel  of  Interest  development  and  tracking;  ISR  planning, 
execution,  and  FED  [define!];  and  MIO  planning,  execution,  and  assessment.  The  key 
data  concerning  this  assessment  area  are  presented  in  Table  14. 

Table  14.  Operations  Performance  Results 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

50 

Knowledge 

Processes: 

Vol 

Development 

Spiral- 1  tools  increased  the  user’s 
ability  to  fuse  vessel,  people,  and 
cargo  data  from  interagency  sources 

operational 

utility 

3 

All  MDA 
Tech 

QRA 

260 

Knowledge 

Processes: 

Vol 

Development 

MIFCPAC  will  require  MDA  tools 
to  discriminate  between  different 
types  or  activities  of  white  vessels, 
e.g.,  US  vs.  foreign  vessels  in 
fisheries. 

features 

3 

All  MDA 
Tech 

Workflow, 

PEW 

48 

Knowledge 

Processes: 

Vol 

Development 

The  lack  of  commonality  between 
nodes  made  the  aggregation  of 
information  difficult.  MDA  Spiral- 1 
did  not  contain  a  central  repository 
of  worldwide  vessel  movement  data. 
Differences  in  node  metadata 
required  additional  research  on  the 
part  of  the  user,  increasing  the  time 
required  to  perform  a  mission. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

All  MDA 
Tech 

QRA 

126 

Knowledge 

Processes: 

Vol 

Development 

Some  users  experienced  situations 
when  there  was  conflicting 
information  when  they  collaborated 
with  users  on  other  systems.  For 
example,  one  time  the  name  of  a 
vessel  was  completely  different  from 
the  name  listed  in  all  other  systems, 
even  though  other  identifying 
features  of  the  vessel  matched.  Also 
sometimes  it  doesn’t  pull  in  all  the 
features  from  Cargolink/  Sealink  that 
it  could,  which  was  frustrating. 
(FAIRGAME) 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

All  MDA 
Tech 

TWOS 

7 

Knowledge 

Processes: 

Vol 

Development 

CMA  is  viewed  as  enhancing  SA 
rapidly.  Ninety  percent  of  users 
either  agreed  or  strongly  agreed  that 
they  could  maintain  awareness  of 
maritime  activity  through  CMA. 
Forty-two  percent  of  users  "agreed," 
26%  "strongly  agreed,"  16% 
"disagreed,"  and  16%  indicated 
"N/A"  that  CMA’s  capability  to 
automatically  acquire  data  on  vessels 
of  interest  from  multiple  databases  is 
faster  than  current  data  acquisition 

operational 

utility 

3 

CMA 

TWOS 

63 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

processes/systems.  Forty-seven 
percent  of  users  "strongly  agreed," 

32%  "agreed,"  16%  were  "n/a"  and 

5%  of  users  "disagreed"  that  CMA’s 
query  capabilities  (search  and 
advanced  search)  save  them 
considerable  time  in  gathering 
information  on  a  vessel  of  interest 
compared  to  current  system 
capabilities.  Fifty-three  percent  of 
users  "strongly  agreed"  and  42% 
"agreed"  that  CMA’s  advanced 
search  capability  to  filter  information 
by  location,  time,  vessel  name, 
vessel  attributes,  etc.,  saves  them 
time  compared  with  having  to  do 
multiple  individual  searches. 

39 

Knowledge 

Processes: 

Vol 

Development 

Participants  thought  conducting 
analysis  on  a  VOI  was  easier  and 
much  faster  when  using  CMA  than 
with  their  previous  system. 

usability 

3 

CMA 

TWOS 

57 

Knowledge 

Processes: 

Vol 

Development 

Forty-seven  percent  of  users 
"agreed,"  37%  "strongly  agreed," 

11%  "n/a,"  and  5%  "disagreed"  that 
sharing  CMA  data  with  other  units 
and  government  agencies  helped 
them  to  gather  and  share  information 
on  vessels  of  interest. 

operational 

utility 

3 

CMA 

TWOS 

115 

Knowledge 

Processes: 

Vol 

Development 

Usability.  Both  CMA  operators 
thought  the  technology  was  easy  to 
learn  and  use.  Both  CMA  users 
strongly  agreed  it  was  easy  to 
develop  and  maintain  situation 
awareness  on  a  vessel  of  interest 
(VOI), 

usability 

3 

CMA 

TRRLOE 

156 

Knowledge 

Processes: 

Vol 

Development 

CMA  features  they  liked:  (1)  Nice 
search  function,  if  a  little  slow;  (2) 
Strongly  agreed  was  able  to  process 
and  maintain  situation  awareness  on 
more  than  one  Vol  at  a  time  more 
easily  than  with  my  previous  system: 
"Especially  with  the  "Map  Viewer" 
and  ‘View  Details"  as  separate 
windows  - 1  love  the  tabbed 
browsing." 

operational 

utility 

3 

CMA 

VBSS 

165 

Knowledge 

Processes: 

Vol 

Development 

Name  variants:  Was  able  to  limit 
searching.  In  addition  was  able  to 
find  specific  vessels  if  I  had  specific 
information. 

operational 

utility 

3 

CMA 

n.a. 

169 

Knowledge 

Processes: 

Vol 

Development 

Associate  data:  This  was  the  most 
helpful  in  that  in  addition  to  the  ship 
information  provided:  Company 
name,  location,  master,  crew  and 
other  information  associated  with  the 
vessel.  Again  as  a  ct  analyst  want  to 
look  at  more  than  just  the  vessel  and 
where  it  is  headed,  want  as  much 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

3 

CMA 

VBSS 

64 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

information  as  possible  that 
associates  the  vessel  with  other 
entities. 

184 

Knowledge 

Processes: 

Vol 

Development 

CMA:  It  was  simply  a  matter  of  click 
on  the  vessel  and  most  of  the 
relevant  information  was  placed 
before  me.  If  I  were  a  ship  tracker 
this  is  great.  In  addition  I  would  be 
able  to  use  the  additional  information 
for  the  CT  mission.  I  am  not  so  much 
interested  in  the  path  of  vessels  hut 
rather  who/  what  is  on  the  vessel." 
(FAIRGAME) 

usability 

3 

CMA 

VBSS 

198 

Knowledge 

Processes: 

Vol 

Development 

CMA:  Thirty-seven  percent  of  users 
"agreed,"  32%  indicated  "N/A,"  26% 
"strongly  agreed,"  and  5%  "strongly 
disagreed"  they  liked  the  ability 

CMA  gives  to  leave  an  "alert"  in  a 
vessel  file  that  can  be  used  by  other 
analysts.  Forty-two  percent  of  users 
indicated  "N/A,"  26%  "agreed,"  21% 
indicated  "disagree,"  and  1 1% 
indicated  "strongly  agree"  that  they 
could  count  on  the  pre-defmed  alerts 
in  CMA  to  detect  conditions  of 
interest  and  that  CMA  pre-defined 
alerts  helped  them  stay  informed 
about  Vols.  Forty-two  percent  of 
users  indicated  "N/A,"  32% 

"agreed,"  11%  "disagreed,"  and  16% 
strongly  agreed  CMA  pre-defined 
alerts  were  understandable. 

alerts 

3 

CMA 

TW08 

235 

Knowledge 

Processes: 

Vol 

Development 

Sixty-eight  percent  of  users  "strongly 
agreed"  and  26%  of  users  "agreed" 
that  CMA’s  capability  to  correlate 
vessel,  cargo,  or  people  information 
automatically  and  store  it  in  a  single 
system  makes  my  job  easier  and 
saves  me  time.  Thirty-seven  percent 
of  users  "strongly  agreed,"  37%  of 
users  "agreed,"  21%  of  users 
indicated  "N/A"  and  5%  of  users 
"disagreed"  that  CMA’s  hypergraph 
display  of  vessel  relationships  with 
other  vessels,  crews,  etc.,  is  useful  in 
rapidly  identifying  related 
information. 

usability 

3 

CMA 

Workflow, 

PEW 

237 

Knowledge 

Processes: 

Vol 

Development 

Sixty-three  percent  of  users  "agreed" 
and  34%  of  users  "strongly  agreed" 
that  CMA’s  display  of  vessel 
attributes  is  useful  in  their  analysis 
of  vessels  of  interest.  Sixty-three 
percent  of  users  "agreed,"  26%  of 
users  "strongly  agreed,"  16%  of 
users  were  "N/A,"  and  5%  of  users 
"disagreed"  that  the  quality  of  CMA 
automated  track  development 

operational 

utility 

3 

CMA 

TW08 

65 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

functions  supports  their  analysis 
tasks. 

239 

Knowledge 

Processes: 

Vol 

Development 

Both  CMA  users  strongly  agreed  it 
was  easy  to  develop  and  maintain 
situation  awareness  on  a  vessel  of 
interest  (VOI),  especially  once 
provided  with  a  watch  list  that 
includes  the  names  of  the  vessels. 
CMA  operators  emphasized  that  it 
was  extremely  easy  to  find  relevant 
information  on  a  VOI  as  long  as  one 
knew  the  name  of  the  vessel; 
otherwise  it  would  be  difficult  due  to 
the  hyper  graph. 

operational 

utility 

3 

CMA 

TRRLOE 

242 

Knowledge 

Processes: 

Vol 

Development 

It  was  easy  for  CMA  users  to 
associate  tipper/  Intel  information 
with  the  correct  VOI.  CMA  showed 
recent  alerts  on  the  vessel  and  made 
it  easy  to  see  the  tippers. 

usability 

3 

CMA 

TRRLOE 

248 

Knowledge 

Processes: 

Vol 

Development 

Forty-two  percent  of  users  "agreed," 
32%  of  users  "strongly  agreed,"  and 
26%  of  users  indicated  "N/A"  that 
their  ability  to  individually  identify 
and  save  analytic  areas  of  interest  in 
CMA  helps  them  in  their  analytic 
work.  Fifty-eight  percent  of  users 
"strongly  agreed"  and  40%  of  users 
"agreed"  that  CMA’s  ability  to 
display  the  track  history  of  vessels  is 
important  to  their  analytic  work. 

operational 

utility 

3 

CMA 

TWOS 

250 

Knowledge 

Processes: 

Vol 

Development 

CMA  -  All  four  users,  surveyed 
during  FAIRGAME,  agreed/ 
strongly  agreed  it  was  easy  to  search 
and  track  a  Vol.  Users  were  mixed 
regarding  whether  they  were  able  to 
process  and  maintain  situation 
awareness  on  more  Vols  then  with 
their  previous  system.  (FAIRGAME) 

usability 

3 

CMA 

VBSS 

33 

Knowledge 

Processes: 

Vol 

Development 

During  FAIRGAME,  users  were 
mixed  (responses  ranged  from 
disagree  to  agree)  regarding  whether 
they  were  able  to  develop  awareness 
of  VOIs  faster  with  CMA  "Again, 
definitely  when  there  was  a  good 
fused  picture  or  a  complicated  track  I 
needed  to  follow.  I  imagine  for  most 
VOIs  it  won’t  be  significantly  faster, 
though."  All  users  agreed/strongly 
agreed  it  was  easy  to  find 
information  on  a  Vol. 

(FAIRGAME) 

usability 

2 

CMA 

VBSS 

42 

Knowledge 
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CMA  users  suggested  that  the 
system  should  include  a  search  pull 
down  in  the  metadata  section  that 
allows  for  the  search  of  cargo, 
making  it  easier  to  narrow  down  the 
hits. 
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128 

Knowledge 

Processes: 

Vol 

Development 

All  users  surveyed  by  NPS  during 
FAIRGAME  viewed  CMA  as  an 
effective  collaboration  tool,  although 
half  had  experienced  conflicting 
information  when  they  collaborated 
with  other  operators  on  a  Vol. 
(FAIRGAME) 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

VBSS 
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Knowledge 
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To  an  extent  it  is  helpful  so  see  a 
fused  picture  of  all  the  Intel  related 
to  a  specific  vessel,  hut  ONLY  when 
this  was  available  (sometimes  there 
just  isn’t  much  more  available  on  a 
vessel  than  what  is  in 
Sealink/SeaWatch)  or  when 
complicated  scenarios  demanded  the 
use  of  CMA  instead  of  a  less 
advanced  tracking  system." 
(FAIRGAME) 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

EC  EMIO 

152 

Knowledge 

Processes: 

Vol 

Development 

Three  of  four  users  surveyed  at 
FAIRGAME  indicated  there  were 
feature  they  wanted  to  perform  with 
the  system  that  they  did  not 
understand.  For  example:  "During 
the  exercise  there  was  a  lot  of 
confusion  about  being  able  to  track 
vessels  that  come  near  a  specific 
vessel  at  any  time  during  its  track  - 1 
still  don’t  think  this  function  is  truly 
possible." 

usability 

2 

CMA 

TW08 

154 

Knowledge 

Processes: 

Vol 
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CMA:  There  were  also  actions  they 
wanted  to  perform  with  the  system 
that  were  not  available,  e.g.,  "I  think 
the  map  could  be  more  user-friendly 
-  as  I  mentioned  earlier  one  should 
be  able  to  use  the  bounding  box  and 
then  the  ruler  without  using  the  box. 
Also,  it  would  be  helpful  if  there  was 
a  way  to  merge  tracks  that  the 
system  lists  as  separate  hits  due  to 
slight  discrepancies  but  are  clearly 
the  same  vessel. 

features 

2 

CMA 

VBSS 

158 

Knowledge 

Processes: 

Vol 
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Other  comments  regarding  human- 
system  interaction  for  CMA:  (1) 
Advanced  search  is  tricky  and  not 
very  intuitive  but  does  the  job;  (2) 
the  map  could  be  more  user-friendly 
-  one  should  be  able  to  use  the 
bounding  box  and  then  the  ruler 
without  using  the  box;  (3)  it  would 
be  helpful  if  there  was  a  way  to 
merge  tracks  that  the  system  lists  as 
separate  hits  due  to  slight 
discrepancies  but  are  clearly  the 
same  vessel.  (FAIRGAME) 

usability 

2 
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PEW 

160 
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View  Details:  Color-coded,  fused 

Intel  was  very  helpful  WHEN  it  was 
available.  I  would  use  CMA  maybe 
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after  consulting  one  or  two  baseline 
programs  first,  otherwise  it’s 
sometimes  just  too  impractical  to 
poke  around  CMA  when  it  is 
running  slowly  or  having  errors. 
However,  if  I  have  a  specific  set  of 
variables  for  a  VOI  (i.e.  past 
locations,  people  on  hoard,  flag,  etc.) 
it  is  a  very  helpful  resource. 

correctness, 

conflicts, 

timeliness 

174 

Knowledge 

Processes: 

Vol 

Development 

CMA  Alerts:  Was  unable  to  set  up 
alerts  and  trip  wires  to  notify  of 
transit.  Information  on  cargo  and 
individuals  on  the  vessel  was 
limited. 

alerts 

2 

CMA 

VBSS 

229 

Knowledge 

Processes: 

Vol 

Development 

Sixty-three  percent  of  users  "strongly 
agreed"  and  37%  of  users  "agreed" 
the  ability  to  view  the  underlying 
data  that  is  used  in  CMA’s  track 
development  is  important  to  them  in 
evaluating  a  track. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

TWOS 

231 

Knowledge 

Processes: 
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Development 

Sixty  percent  of  users  "strongly 
agreed"  and  42%  of  users  "agreed" 
that  CMA’s  ability  to  automatically 
identify  differences  in  a  vessel’s 
reported  name,  MMSI,  IMO, 
SCONUM,  call  sign,  emitter 
parametrics,  cargo  or  crew  assists 
them  in  their  analysis. 

operational 

utility 

2 

CMA 

TRRLOE 

246 

Knowledge 

Processes: 

Vol 

Development 

CMA  -  All  four  users,  surveyed 
during  FAIRGAME,  agreed/ 
strongly  agreed  it  was  easy  to 
develop  and  maintain  situation 
awareness  on  a  Vol.  Users  were 
mixed  (ranged  from  neutral  to 
strongly  agree)  regarding  whether  it 
was  easy  to  (1)  find  information  on  a 
Vol;  (2)  associate  tipper/  intel 
information  with  the  correct  Vol; 
and  (3)  conduct  analysis  on  a  Vol. 
(FAIRGAME) 

operational 

utility 

2 

CMA 

VBSS 

252 
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Processes: 

Vol 

Development 

CMA  -  Users  were  mixed  regarding 
whether  it  was  easy  to  (1)  collaborate 
on  a  Vol  and  (2)  cross  reference 
tracks  being  processed  by  operators 
using  other  systems.  Users  were 
mixed  regarding  how  easy  CMA 
made  it  for  them  to  collaborate  about 
the  same  information  with  users  on 
other  systems.  (FAIRGAME)  Yet, 
they  all  agreed  it  as  easy  to  cross 
reference  information  on  Vols  with 
operators  using  other  systems. 

usability 

2 
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CMA  -  The  envisioned  CONOPS  for 
future  MDA  will  require  training  and 
familiarization  with  the  new 
procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly 

training 
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collaborate  with  others,  with  the 
exception  of  perhaps  shared  watch 
areas;"  and  (2)  "During  the  exercise 
there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come 
near  a  specific  vessel  at  any  time 
during  its  track  - 1  still  don’t  think 
this  function  is  truly  possible." 

301 

Knowledge 

Processes: 

Vol 

Development 

CMA  -  The  envisioned  TTP/SOP  for 
future  MDA  will  require  training  and 
familiarization  with  the  new 
procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly 
collaborate  with  others,  with  the 
exception  of  perhaps  shared  watch 
areas;"  and  (2)  "During  the  exercise 
there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come 
near  a  specific  vessel  at  any  time 
during  its  track  - 1  still  don’t  think 
this  function  is  truly  possible." 

training 
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Tactical  EMIO  System  (TES)  - 
While  mobility  of  the  Tactical  EMIO 
Device  (TED)  was  a  clear  advantage, 
enabling  the  collection  of  data  from 
multiple  locations  within  the  vessel, 
one  limitation  noted  was  that  the 

TED  must  be  within  the  vicinity  of 
the  Tactical  EMIO  Maritime  PC 
(TEMP)  to  download  the  data 
captured.  Although  wireless,  the 

TED  devices  were  required  to  be  in 
the  vicinity  of  the  TEMP  in  order  to 
download  data  captured  during  the 
boarding.  The  radio  frequency  (RF) 
signals  were  not  strong  enough  to 
transmit  data  when  team  members 
were  below  decks.  The  Maritime 
BGAN  EMIO  Terminal  (MEET) 
device,  in  turn,  failed  to  transfer  data 
due  to  environmental  issues  and 
weak  RF  Signal  range  of  the 
commercial  satellite.  Contractors 
eventually  departed  the  target  vessel 
and  drove  inland  with  the  TEMP  and 
MEET  device  to  acquire  a  stronger 
signal.  The  MEET  link  was  then 
acquired  and  successfully 
transmitted  data  from  the  TEMP 
device.  Contractors  asserted  that  the 
satellite  connectivity  will  not  be  a 
concern  in  the  current  AOR.  Also, 
the  boarding  officer  was  not  able  to 
demonstrate  the  transfer  of  data  via 
the  Maritime  EGAN  EMIO 

Terminal  (MEET)  due  to  the  satellite 
connectivity.  (VESS  School) 
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24 

Knowledge 
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Vol 

Development 

The  user  had  no  knowledge  of 
foreign-to-foreign  cargo  sharing 
agreements  or  the  amount  of  cargo 
data  actually  available,  which 
affected  the  ability  to  build  and 
evaluate  cargo  queries. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

Global 

Trader 

QRA 

120 

Knowledge 

Processes: 
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Development 

Usability.  The  operator  felt  that 
Google  Earth  was  easy  to  learn,  and 
use,  finding  no  difficulty  in  using  all 
of  its  features.  The  system  was  a 
quick  geospatial  reference  for 
developing  awareness  on  a  VOI  as 
well  as  finding  information  on  them. 

usability 

3 

Google 

Earth 

QRA 

244 

Knowledge 

Processes: 

Vol 

Development 

Conducting  analysis  on  a  VOI  with 
Google  Earth  was  easy,  specifically 
for  acquiring  their  last  position; 
however  the  VOLPE  AIS  data  was 
limited,  and  not  useful  for  in-depth 
analysis  or  for  providing  tracking 
data. 

usability 

3 

Google 
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91 
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Development 

Respondents  mostly  agreed  that 
master's  capability  to 
automatically  acquire  data  on  vessels 
of  interest  is  superior  to  current  data 
acquisition  processes.  MASTER’S 
track  analysis  and  graphing 
capabilities  were  among  its 
strengths,  but  its  archive  database 
was  often  slow  to  respond  to  queries. 
MASTER  was  easy  to  use  when 
creating  new  vessel  tracks. 

operational 

utility 

3 

MASTER 

TWOS 
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Its  user  interface  could  be  improved 
by  making  the  vessel  of  interest 
category  more  salient  and  improving 
the  mapping  program. 

usability 
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Respondents  differed  on  whether 
MASTER  reduced  their  workload 
compared  to  current  methods.  1 1  of 
the  18  responses  to  this  item  agreed 
(4  strongly)  that  MASTER  reduced 
their  workload,  but  the  other  7 
responses  disagreed.  Comments 
supporting  MASTER  noted  its 
analytical  capabilities  and  ability  to 
streamline  multiple  activities  into 
one  system,  saving  significant  time. 
One  respondent  noted  that  MASTER 
reduced  his  workload  by  25%,  but 
would  be  even  more  effective  if  the 
ship’s  beneficiary  country  was 
included  in  the  query  info. 

Dissenting  comments  included: 
MASTER  was  not  easily  accessible; 
it  cannot  locate  vessels  in  real  time 
or  find  a  first-time  vessel  with  no 
history;  its  controls  are  awkward; 
and  the  computers  were  slow. 
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Criticisms  noted  that  smaller  vessels 
with  SCONUMS  are  hard  to  detect, 
the  archive  database  is  slow,  some 
data  are  not  in  real  time,  a  48-hour 
history  is  sometimes  insufficient. 
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processing 
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timeliness 
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MASTER:  Respondents  agreed 
MASTER  is  very  useful  for  maritime 
analysis.  The  track  analysis  and 
plotting  features  are  excellent  tools, 
although  the  large  amount  of 
information  MASTER  provides  may 
hamper  a  rapid  analysis. 

Respondents  also  agreed  MASTER’S 
capability  to  integrate  multi-source 
intelligence  and  information  with 
fused  SuperTrack  data  is  valuable.  It 
saves  time,  facilitates  track  fusion 
and  analysis,  and  centralizes  vessel 
data. 

operational 

utility 
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MASTER:  Some  users  questioned 
the  completeness  or  accuracy  of  the 
information  in  MASTER’S  database. 
Several  users  used  SeaLink  to  find 
information  on  a  port  and  to 
correlate  old  ship  names  with  new 
ship  names.  One  user  utilized  the 
PMIC  web  page  in  attempting  to 
determine  the  destination  for  a 
particular  ship.  Two  users  preferred 
ASA  over  MASTER  because  it  is 
both  faster  and  easier  to  use,  not 
because  it  is  more  accurate  than 
MASTER.  Respondents  differed  on 
whether  MASTER  processes 
multiple-source  intelligence  faster 
than  current  methods.  10  of  the  21 
responses  to  this  item  agreed  that  it 
did,  but  the  other  1 1  disagreed. 
Criticism  focused  on  MASTER’S 
slowness,  especially  its  slow  query 
responses  and  map  updates.  Current 
tools  (GALE,  Sealink)  were 
described  as  being  much  faster  than 
MASTER.  Another  drawback  is 
master’s  current  limit  of  one 
vessel  per  query. 
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MASTER:  Respondents  agreed  that 
MASTER  is  easy  to  use  for  creating 
tracks  for  vessels  of  interest, 
although  one  respondent  noted  that 
vessel  tracks  are  archived  too 
quickly  to  use  for  tracking  VOIs. 
Recommended  user  interface 
improvements  included:  make  the 
vessel  of  interest  category  more 
salient  in  current  vessel  details. 
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improve  the  mapping  program,  and 
add  the  ability  to  take  tracks  from 
MASTER  and  transfer  them  into 
GALE  to  improve  EIE  tracking.  One 
respondent  wanted  to  be  able  to 
export  track  data  to  Excel  to  better 
compare  tracks  and  to  compile  data 
more  effectively. 

137 

Knowledge 

Processes: 

Vol 

Development 

Maritime  Domain  Awareness  Data 
Sharing  Community  of  Interest 
(MDA  DS  COI).  Feedback  indicates 
that  usability  was  relatively  low. 

Users  were  able  to  identify  suspect 
VOIs,  however,  it  was  difficult.  One 
observer  stated  "This  user  interface 
needs  major  improvement  with 
respect  to  correlating  data 
relationships  between  popup 
windows  and  the  map,  and  the  track 
data  playback  function.  The  process 
requires  the  operator  to  bounce 
between  track  data  popup 
spreadsheets  and  layer  tools  lists. 

The  operators  had  to  manually 
document  record  track  identifier 
numbers  to  compare  data  list  on 
other  windows.  This  process  is  prone 
to  errors  as  demonstrated  during  this 
event." 

usability 

2 

MDADS 

COI 

TWOS 

59 

Knowledge 

Processes: 

Vol 

Development 

Maritime  Integrated  Domain 
Awareness  Solution  (MIDAS).  -All 
three  operators  reported  that  MIDAS 
enhanced  their  ability  to  investigate 
concerns  regarding  VOIs  or  persons 
of  interest  and  to  understand  why  a 
vessel  is  suspicious  /  threatening. 

Two  of  the  3  said  it  enhanced  their 
ability  to  display  a  user-defined 
picture  of  the  operational 
environment,  and  that  it  provides 
access  to  relevant  commercial  and 
law  enforcement  data.  All  agreed 
that  what  is  'relevant  data'  in  military 
operations  is  relative  and 
changeable;  2  of  3  agreed  that 

MIDAS  threat  picture  is  able  to  be 
set  to  show  relevance  for  a  particular 
military  situation.  Operators 
generally  agreed  that  they  needed  to 
have  MDA  at  both  classified  and 
unclassified  levels;  3  suggested  that 
MIDAS  would  be  best  for  UNCLAS 
data,  and  2  others  noted  the  need  to 
integrate  the  unclassified  data  'up' 
with  the  classified  data.  MIDAS  data 
was  mentioned  as  possibly 
complementing  GCCS  data  for  a 
fuller  operational  picture. 

operational 
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PANDA:  Regarding  predicting 
deviation,  most  users  (85%) 
responded  that  detecting  deviation  is 
a  useful  capability  and  all  users 
responded  that  alerting  for  deviation 
is  a  useful  capability.  Most  users 
responded  that  the  capability  of 
classifying  alerts  into  categories 
(86%)  and  that  PANDA’ s 
classification  of  alerts  (86%)  were 
useful.  User  responses  were  mixed 
about  whether  PANDA’ s 
explanation  for  the  deviation  was 
useful.  Information  that  users  would 
communicate  to  others  about 
deviations  included  %  likelihood  a 
vessel  would  deviate,  context 
information  such  as  ship 
parts/ spares/repair,  and  crew 
change/illness  (currently  not 
available  in  PANDA),  other  vessels 
in  the  area,  whether  a  vessel  maybe  a 
potential  threat  or  information  for  a 
task. 
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Predictive  Analysis  for  Naval 
Deployment  Activity  (PANDA): 

Most  users  indicated  that  they 
information  provided  in  PANDA 
was  easy  to  understand.  Eighty-six 
percent  agreed  (others  were  neutral) 
it  was  easy  to  understand  the  vessel 
list  details;  93%  agreed  (1  user 
disagreed)  the  use  of  color  associated 
with  deviation  types  made  sense;  all 
users  agreed  it  was  easy  to 
understand  the  meaning  of  the  color 
coding;  86%  agreed  (others 
disagreed)  that  it  was  easy  to 
understand  the  meaning  of  the  labels 
used  for  deviation;  75%  agreed  (1 
was  neutral)  PANDA  made  it 
apparent  that  a  vessel’s  behavior  was 
similar  to  previous  behavior. 

Regarding  normalcy:  All  users 
indicated  that  the  PANDA  was 
effective  at  representing  the  normal 
behaviors  of  vessels  in  the  PANDA 

survey. 
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PANDA:  Regarding  deviation:  All 
users  responded  that  PANDA  was 
effective  at  alerting  for  deviations 
and  that  PANDA’ s  explanation  for 
the  deviation  was  reasonable  and 
made  sense.  Most  users  indicated 
that  it  was  easy  to  understand  the 
explanation  for  the  deviation  (67% 
and  83%  strongly  agreed;  others 
were  neutral).  Regarding  prediction: 
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Sixty-seven  percent  of  users  (67%) 
responded  that  PANDA  was 
effective  at  representing  predicted 
vessel  behavior  (others  were  neutral 
and  disagreed). 

219 

Knowledge 
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Development 

PANDA:  All  12  users  indicated 
PANDA  would  have  a  positive  effect 
on  their  situation  awareness.  Users 
pointed  to  a  variety  of  PANDA 
features  that  would  support 
improved  SA,  including  an  increase 
in  understanding  of  normalcy  and 
anomalous  vessel  behavior,  alerting 
to  behavior  that  would  otherwise  be 
missed,  prediction  related  to  ships 
inbound  to  US  ports,  understanding 
of  vessel  history,  and  potential 
support  for  "proactive"  analysis. 
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Tripwire  Contextual  information  on 
vessels  of  interest  and  persons  of 
interest  is  outstanding. 
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The  majority  of  users  reported  liking 
the  ability  that  CMA  gives  them  to 
leave  an  alert  in  a  vessel  file  that  can 
be  used  by  other  analysts.  Users  also 
reported  being  able  to  rely  on  pre¬ 
defined  alerts  to  stay  informed  about 
vessels  of  interest.  Regarding  the 
timely  attribute,  users  reported  that 
query  functions  saved  significant 
time.  Specifically,  it  was  helpful  to 
be  able  to  filter  by  location,  time, 
vessel  name  and  vessel  attributes. 

alerts 
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Bounding  Box:  Was  able  to  mark 
and  track  areas  w/  precision 

usability 

3 

CMA 

VBSS 

167 

Knowledge 

Processes: 

Vol  Tracking 

Tracking  data:  Was  able  to  see  the 
exact  posits  in  the  past  and 
corroborate  with  the  reporting 

misc 

3 

CMA 

TRRLOE 

13 

Knowledge 

Processes: 

Vol  Tracking 

CMA  provided  searchable  data  on 
vessel  location,  cargo,  and  people 
but  there  were  gaps  in  track  data,  a 
lack  of  commonality  of  data  between 
nodes,  and  insufficient  information 
on  data  sources. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

QRA 

17 

Knowledge 

Processes: 

Vol  Tracking 

CMA:  On  several  occasions  a  query 
of  current  vessel  position  returned 
multiple  vessel  positions,  each  with 
different  metadata  (time,  cargo,  and 
flag)  attached. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

2 

CMA 

QRA 

19 

Knowledge 

Processes: 

Vol  Tracking 

Users  noted  a  general  inconsistency 
of  the  CMA  vessel  database  between 
nodes  within  the  same  AOR.  An 

data  & 

processing 

completeness. 

2 

CMA 

QRA 

74 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

analysis  was  performed  on  the 
commonality/consistency  of  track 
data  held  by  the  NAVCENT  and 
PACFLT  CMA  systems  within  a  5 
by  10  deg  geographic  region  in  the 
Indian  Ocean.  Although  97%  of  the 
tracks  were  held  by  both  systems, 
significant  track  metadata 
differences  were  observed  in  1 5%  of 
the  tracks,  and  track  history  data 
were  only  51%  common  (77% 
common  for  tracks  going  from  the 
NAVCENT  AOR  to  the  PACFLT 
AOR,  and  29%  common  for  tracks 
going  in  the  other  direction).  Some 
track  history  data  in  one  site’s  CMA 
were  not  in  the  other  site’s  CMA. 

correctness, 

conflicts, 

timeliness 

227 

Knowledge 

Processes: 

Vol  Tracking 

Based  on  survey  responses  (n=19), 
CMA  provides  users  with  data  that  is 
relevant  and  helpful  to  their  analysis 
of  conditions  of  interest  and  CMA 
facilitated  the  reliable  and  timely 
analysis  of  maritime  information. 

With  regards  to  reliability,  users 
reported  that  it  is  important  to  be 
able  to  view  underlying  data 
provided  by  CMA  in  evaluating 
tracks.  Regarding  timeliness,  users 
reported  that  being  able  to  access 
data  from  one  source  (CMA)  was 
faster  than  existing  procedures. 

Users  also  stated  that  it  made  their 
job  easier  to  be  able  to  correlate  and 
store  information  in  a  single  system. 
Finally,  63%  of  users  strongly 
agreed  that  response  time  was 
adequate. 

usability 

2 

CMA 

TWOS 

15 

Knowledge 

Processes: 

Vol  Tracking 

The  user’s  ability  to  monitor  vessel, 
person,  and  cargo  data  was  severely 
degraded  by  gaps  in  track  data 
coverage.  When  a  node’s  CMA 
server  was  down,  or  data  was  not 
transmitted,  the  data  not  received 
was  not  recoverable.  The  Naval 
Research  Laboratory  (NRL)  limited 
the  National  Technical  Means 
(NTM)  data  source  input  to  CMA  to 

14  hours  per  day  and  filtered  the  data 
that  was  provided.  This  resulted  in  a 
gap  of  data  which  had  a  negative 
impact  across  all  AORs. 

data  & 

processing 

completeness, 

correctness, 

conflicts, 

timeliness 

1 

CMA 

QRA 

98 

Knowledge 

Processes: 

Vol  Tracking 

MASTER:  Respondents  agreed  they 
could  track  vessels  as  needed. 
MASTER  does  a  very  good  job  of 
pulling  vessel  track  history,  provides 
named  data  that  current  systems  do 
not,  and  facilitates  vessel  tracking 

operational 

utility 

3 

MASTER 

TWOS 

75 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

with  its  graphical  display  and  ability 
to  compare  tracks. 

216 

Knowledge 

Processes: 

Vol  Tracking 

One  user  stated  that  MASTER 
completes  tasks  in  20  minutes  that 
would  take  a  GALE  Lite  user  10 
hours  because  GALE  Lite  users  have 
to  manually  stitch  tracks. 

speed 

3 

MASTER 

Workflow, 

PEW 

258 

MIO: 

Execution 

MIO  situation  awareness  was 
maintained  at  rear  units. 

operational 

utility 

3 

E-MIO 

Wireless 

EC  EMIO 

180 

MIO: 

Execution 

Tactical  EMIO  System  (TES)  - 
Observations  of  the  Maritime  BGAN 
EMIO  Terminal  (MEET) 
demonstrated  that  (1)  its  design  was 
very  rugged  and  seemed  to  prevent 
water  intrusion;  and  (2)  the  MEET 
device  was  not  light  enough  to  be 
carried  on  a  VESS  person.  The 
student  user’s  felt  that  (1)  the 
equipment  should  integrate  the 
camera  along  with  the  biometric 
capability;  (2)  the  Tactical  EMIO 
Device  (TED)  had  too  many 
functions  to  organize  the  data;  (3) 
they  would  prefer  required  less  steps 
to  configure  the  system;  (4)  the 

EMIO  Gear  was  very  mobile;  and 
(5)  the  graphic  user  interface  was 
intuitive  and  simple  to  use.  (VESS 
School) 

usability 

2 

E-MIO 

Wireless 

TRRLOE 

194 

MIO: 

Execution 

Tactical  EMIO  System  (TES)  - 
Training  for  use  of  the  Tactical 

EMIO  System  (TES)  was  conducted 
at  the  VBSS  School  in  San  Diego 
and  testing  was  planned  for  two 
sessions  at  the  conclusion  of  two 
courses  (June  and  July).  As  is  so 
often  the  case,  the  10  student  system 
users  felt  they  did  not  receive 
enough  training,  and  in  particular, 
time  to  practice  operating  the 
system. 

training 

2 

E-MIO 

Wireless 

VBSS 

196 

MIO: 

Execution 

Observing  the  operation  of  the 

Tactical  EMIO  Device  (TED)  in  the 
field  indicated  the  students  would 
require  additional  training  to  fully 
operate  the  TED.  However,  the 
students  were  able  to  use  the  features 
to  capture  required  data.  Students 
asked  many  questions  during  the 
training  which  provided  useful 
feedback  to  learn  the  perspective  of 
the  user.  (VBSS  School) 

training 

2 

E-MIO 

Wireless 

Workflow, 

PEW 

278 

MIO: 

Execution 

Tactical  EMIO  System  (TES)  - 
Students  were  not  granted  enough 
time  to  practice  operating  all  three 
systems  (TED,  TEMP,  and  MEET). 
Observing  the  operation  of  the 

training 

2 

E-MIO 

Wireless 

VBSS 

76 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

Tactical  EMIO  Device  (TED)  in  the 
field  indicated  the  students  would 
require  additional  training  to  fully 
operate  the  TED. 

86 

MIO: 

Execution 

Tactical  EMIO  System  (TES)  - 
While  mobility  of  the  Tactical  EMIO 
Device  (TED)  was  a  clear  advantage, 
enabling  the  collection  of  data  from 
multiple  locations  within  the  vessel, 
one  limitation  noted  was  that  the 

TED  must  be  within  the  vicinity  of 
the  Tactical  EMIO  Maritime  PC 
(TEMP)  to  download  the  data 
captured.  Although  wireless,  the 

TED  devices  were  required  to  be  in 
the  vicinity  of  the  TEMP  in  order  to 
download  data  captured  during  the 
boarding.  The  radio  frequency  (RF) 
signals  were  not  strong  enough  to 
transmit  data  when  team  members 
were  below  decks.  The  Maritime 
BGAN  EMIO  Terminal  (MEET) 
device,  in  turn,  failed  to  transfer  data 
due  to  environmental  issues  and 
weak  RF  Signal  range  of  the 
commercial  satellite.  Contractors 
eventually  departed  the  target  vessel 
and  drove  inland  with  the  TEMP  and 
MEET  device  to  acquire  a  stronger 
signal.  The  MEET  link  was  then 
acquired  and  successfully 
transmitted  data  from  the  TEMP 
device.  Contractors  asserted  that  the 
satellite  connectivity  will  not  be  a 
concern  in  the  current  AOR.  Also, 
the  boarding  officer  was  not  able  to 
demonstrate  the  transfer  of  data  via 
the  Maritime  BGAN  EMIO 

Terminal  (MEET)  due  to  the  satellite 
connectivity.  (VBSS  School) 

connectivity 

1 

E-MIO 

Wireless 

VBSS 

122 

MIO: 

Execution 

Tactical  EMIO  System  (TES)  - 
Although  wireless,  the  TED  devices 
were  required  to  be  in  the  vicinity  of 
the  TEMP  in  order  to  download  data 
captured  during  the  boarding.  The 
radio  frequency  (RF)  signals  were 
not  storing  enough  to  transmit  data 
when  team  members  were  below 
decks.  (VBSS  School) 

connectivity 

1 

E-MIO 

Wireless 

TRRLOE 

224 

MIO: 

Execution 

There  were  no  positions  provided 
with  the  latent  print  messages  and 
for  the  cave  collection,  there  was  no 
SeekID  Global  Positioning  System 
(GPS)  reception  and  therefore  no 
position  was  included  in  the 
biometric  messages. 

features 

1 

E-MIO 

Wireless 

QRA 

77 


6.2  Warfighter  Performance  Results 


The  assessment  area  “warfighter  performanee”  addresses  the  eapability  of  warfighters  to 
conduct  MDA  missions. 

No  findings  bore  directly  on  this  assessment  area. 

6.3  Organization/Guidance  Results 

The  assessment  area  “organization/guidance”  concerns  the  alignment  of  organizations  to 
MDA  missions,  the  alignment  of  MDA  processes  or  tasks  with  those  specified  in  the 
MHQ  with  MOC  process  architecture  ([Jared:  citation]),  the  alignment  of  international 
and  inter-organization  agreements  with  MDA  mission  requirements,  and  the  availability 
of  CONOPS,  TTPs,  SOP,  and  standing  orders  for  MDA  missions.  The  key  data 
concerning  this  assessment  area  are  presented  in  Table  15. 


Table  15,  Organization/Guidance  Results 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

295 

Agreements: 

Information 

Sharing 

Information  from  a  stand-alone 
MAGNET  system  was  not  available 
due  to  sharing  agreements 
concerning  U.S.  person’s 
information. 

constraints 
imposed  by 
policies  and 
agreements 

2 

MAGNET 

QRA 

296 

Agreements: 

Information 

Sharing 

MIDAS:  Respondents 
acknowledged  that  there  is 
currently  a  need  to  collaborate  with 
other  agencies,  coalition  members 
and  non-traditional  partners,  and 
most  felt  that  MIDAS  would  help  to 
facilitate  that  collaboration; 
however,  2  mentioned  that  sharing 
sensitive  or  classified  information 
might  be  a  problem. 

constraints 
imposed  by 
policies  and 
agreements 

2 

MIDAS 

TWOS 

299 

Guidance: 

CONOPS 

CMA  -  The  envisioned  CONOPS 
for  future  MDA  will  require 
training  and  familiarization  with  the 
new  procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly 
collaborate  with  others,  with  the 
exception  of  perhaps  shared  watch 
areas;"  and  (2)  "During  the  exercise 
there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that 
come  near  a  specific  vessel  at  any 
time  during  its  track  - 1  still  don’t 
think  this  function  is  truly 
possible." 

training  for 
CONOPS 

2 

CMA 

VBSS 

302 

Guidance: 

TTP/SOP 

CMA  -  The  envisioned  TTP/SOP 
for  future  MDA  will  require 
training  and  familiarization  with  the 

training  for 

TTP 

2 

CMA 

VBSS 

78 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

new  procedures  as  evidenced  by  the 
following:  (1)  "Have  yet  to  truly 
collaborate  with  others,  with  the 
exception  of  perhaps  shared  watch 
areas;"  and  (2)  "During  the  exercise 
there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that 
come  near  a  specific  vessel  at  any 
time  during  its  track  - 1  still  don’t 
think  this  function  is  truly 
possible." 

290 

MDA 

Compatibility: 

Organization 

Alignment 

ONI  expresses  concern  about  lack 
of  lack  of  re-engineering  of 
processes  and  training. 

coordination 
of  MDA  and 
ONI  processes 

2 

All  MDA 
Tech 

TWOS 

304 

MDA 

Compatibility: 

Organization 

Alignment 

ONI  is  conducting  process  analyses 
concerning  intel  analysis.  It  is  not 
clear  that  this  effort  is  synchronized 
with  the  MDA  technology  effort. 

coordination 
of  MDA  with 
ONI  processes 

2 

All  MDA 
Tech 

Workflow, 

PEW 

276 

MHQ/MOC 

Compatibility: 

Organization 

Alignment 

The  NPS  assessment  team  is 
concerned  that  MDA  TTPs  need  to 
be  sufficiently  flexible  to 
accommodate  differences  between 
COCOMS. 

customization 
to  local  needs 

2 

All  MDA 
Tech 

TWOS 

303 

MHQ/MOC 

Compatibility: 

Process 

Alignment 

Second  Fleet, 

NAVNETWARCOM,  and  others 
concur  that  the  MDA  workflow 
aligns  with  the  MHQ  w/MOC 
process  architecture  as  of  early 

2008.  (See  Process  Alignment 
Workshop). 

alignment  of 
MDA  and 

MHQ  w/MOC 
processes 

3 

All  MDA 
Tech 

TWOS 

79 


6.4  System  Supportability  and  Readiness 


The  assessment  area  “System  Supportability  and  Readiness”  coneems  operations  and 
maintenance  required  to  ensure  the  readiness  of  forces  using  Spiral- 1  systems.  This 
assessment  is  made  by  PEO  C4I.  However,  several  data  points  bore  on  these  general 
concerns  (see  Table  16).  Findings  specific  to  training  are  presented  in  assessment  area 
5.3.4;  Warfighter  Acceptance:  System  Training. 


Table  16.  System  Supportability  and  Readiness  Results 


Item 

ID 

Area 

Item 

Comments 

Score 

Technology 

Venue 

273 

System 
Supportability 
and  Readiness 

The  NFS  assessment  team  is 
concerned  that  variance  in  IT 
environments  will  complicate 
installation. 

infrastructure 

requirements 

2 

All  MDA 
Tech 

QRA 

274 

System 
Supportability 
and  Readiness 

NORTHCOM  is  concerned  that 
its  IT  facilities  may  not  be  able  to 
accommodate  additional 
technology. 

infrastructure 

requirements 

2 

All  MDA 
Tech 

Workflow, 

PEW 

275 

System 
Supportability 
and  Readiness 

ONI  is  concerned  that  additional 
power  capacity  may  be  needed  to 
run  additional  technology. 

infrastructure 

requirements 

2 

All  MDA 
Tech 

Workflow, 

PEW 

282 

System 
Supportability 
and  Readiness 

ONI  is  concerned  with 
adequately  staffing  for  MDA 
technologies. 

staff  size  and 
competency 

2 

All  MDA 
Tech 

Workflow, 

PEW 

283 

System 
Supportability 
and  Readiness 

NAVCENT  is  concerned  about 
manning  and  maintaining  MDA 
technologies. 

staff  size  and 
competency 

2 

All  MDA 
Tech 

n.a. 

284 

System 
Supportability 
and  Readiness 

CINCPACFLT  and  NAVCENT 
state  that  they  will  require 
additional  staff  to  operate  and 
maintain  the  technologies. 

staff  size  and 
competency 

2 

All  MDA 
Tech 

Workflow, 

PEW 

288 

System 
Supportability 
and  Readiness 

CINCPACFLEET  is  concerned 
about  the  feasibility  of  learning, 
using,  and  maintaining  new 

MDA  technologies  given  that  it 
has  a  small  Intel  unit. 

staff  size  and 
competency 

2 

All  MDA 
Tech 

Workflow, 

PEW 

80 


7.0  Appendix  A:  Workflow  Architecture  Results 


The  Naval  Postgraduate  Sehool  (NPS),  under  eontraet  to  OPNAV  N3/N5,  eondueted  an  analysis 
of  eurrent,  MDA-related  workflow  in  seleeted  venues  and  identified  potential  eoneerns  about  the 
impaet  of  Spiral- 1  teehnologies  on  that  workflow.  This  appendix  reports  on  the  methodology, 
findings,  and  reeommendations  of  that  study.  Detailed  results  of  that  effort  were  reported  1 
Mareh  2008  (see  report  NPS-IS-08-002). 


7.1  Method 


The  objeetive  of  the  workflow  analysis  was  to  define  the  eurrent  -  or  “as  is”  -  flow  of  MDA 
tasks  in  a  representative  sample  of  organizations,  and  to  eapture  issues  of  eoneern  to  those 
organizations  regarding  MDA  Spiral- 1  teehnologies. 

The  NPS  researeh  team  elieited  information  about  workflow  and  teehnology-related  issues  in 
interviews^  with  CINCPACFLT,  MIFCPAC,  NMIC/ONI,  and  NAVCENT.  (Additional  data 
were  gathered  in  a  Proeess  Engineering  Workshop  and  a  Proeess  Alignment  Workshop,  both 
reported  in  a  subsequent  appendix).  We  eondueted  two  types  of  analyses  using  these  data. 

1 .  A  qualitative  analysis  was  performed  to  identify  areas  of  eoneern  and  generate 
reeommendations  eoneerning  Spiral-1  teehnologies  and  proeess. 

2.  A  proeess  analysis  was  performed  to  define  (1)  eurrent  MDA  tasks,  (2)  the  entities  that 
exeeute  them,  (3)  the  preeedenee  relationships  (or  flow)  between  those  tasks,  and,  when 
possible,  (4)  the  media  used  to  communieate  between  tasks,  (5)  the  produets  of  those 
tasks,  and  (6)  the  potential  applieation  points  for  Spiral- 1  teehnologies.  Representatives 
of  more  than  20  organizations  reviewed  and  revised  the  workflow  produets"^  in  two 
workshops  -  a  Proeess  Engineering  Workshop  and  Proeess  Alignment  Workshop 
reported  in  seetion  8.0. 

Reeommendations  from  this  researeh  were  developed  after  eompleting  the  field  researeh  and 
workshop,  and  are  reported  at  the  end  of  seetion  8.0. 


^  The  opinions  of  the  informants  may  not  accurately  represent  the  position  of  their  organizations. 

The  MDA  workflow  is  documented  in  report  NPS-IS-08-002,  available  upon  request.  NAVNETWARCOM 
(through  its  contractors  WBB  Inc.  and  Booz  Allen  Hamilton)  elaborated  these  data  to  create  DoDAF  OV-6c 
diagrams  of  MDA  activities. 
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7.2  Findings 


Qualitative  findings  from  the  interviews  are  presented  here  in  detail.  They  are  organized  by  the 
data  eolleetion  event  in  whieh  they  were  eollected:  (1)  interviews  with  CINCPACFLT, 
MIFCPAC,  NMIC/ONI,  and  NAVCENT,  and  (2)  a  Proeess  Engineering  Workshop  attended  by 
representatives  from  a  variety  of  organizations. 


7.2. 1  Interviews  at  CINCPACFLT 

Representative  of  the  NPS  research  team  met  with  the  CPE  N2  MOC  Director  and  overall  MDA 
Eead  for  N2  during  the  period  22-28  October  2007  (Kurtz,  2007).  The  interview  was  informal, 
and  focused  on  the  organizational  and  mission  environment  for  Trident  Warrior  technologies. 

Interviewees  raised  several  issues  related  to  MDA  technologies: 

•  The  MOC  to  be  stood  up  by  3 1  Jan  2008  will  have  a  traditional  organization.  It  will  not 
include  ONA;  that  function  will  be  executed  by  N2.  The  intelligence  staff  is  quite  small 
(CPE  N2  currently  has  2  Information  System  specialists  and  one  is  an  E-9),  though  there 
is  an  effort  underway  to  extend  the  human  resources  by  combining  the  CPE  Intel  Watch 
with  the  PACOM  JIOC.  Given  the  small  size  of  its  intelligence  unit,  CPE  is  concerned 
about  the  feasibility  of  learning,  using,  and  maintaining  new  MDA  technologies. 

•  CPE  does  not  need  to  maintain  awareness  of  white  shipping  for  its  routine  operations, 
though  the  capability  is  seen  as  potentially  useful.  However,  CPE  does  need  this 
capability  to  support  one,  highly  complex  OPEAN  (intentionally  unnamed,  here).  Thus, 
use  of  Spiral- 1  technologies  may  be  sporadic  or  localized  to  very  few  staff. 


7.2.2  Interviews  at  MIFCPAC 

Representative  of  the  NPS  research  team  met  with  representatives  of  MIECPAC  7  January  2008 
(MacKinnon  &  Hutchins,  2008).  The  interviews  were  informal.  They  focused  on  how  MDA  is 
viewed  and  accomplished  by  the  Coast  Guard  at  MIECPAC  with  special  attention  to  current  and 
potential  usage  of  Spiral- 1  technologies. 

The  interviewees  raised  several  issues  related  to  MDA  Spiral- 1  technologies: 

•  MIECPAC  is  responsible  for  all  vessels  approaching  the  US  from  continents  except 
Europe.  The  organization  provides  considerable  support  to  CINCPACEET,  which  has  a 
small  staff  (see  section  0).  However,  MIECPAC  is  focused  on  the  Coast  Guard  mission, 
which  concerns  both  terrorism  and  regulatory  issues  such  as  fisheries  and  pollution.  Thus, 
its  use  of  Spiral- 1  tools  may  be  unusual.  Eor  example,  MIECPAC  may  need  alerts  that 
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discriminate  reliably  between  loitering  in  fisheries  by  (1)  Ameriean  vessels  and  (2) 
potentially  illegal  foreign  fishing  vessels. 

•  MIFCPAC  sees  value  in  seleeted  MDA  teehnologies.  It  is  already  using  CMA  to  support 
analyses  (sueh  as  the  fisheries  analysis,  above),  and  it  sees  promise  in  Google  and  Global 
Trader.  MIFCPAC  argues  that  FASTC2AP  may  not  be  "viable"  for  its  uses. 


7.2.3  Interviews  at  NMIC  /  ONI 

Representatives  of  the  NPS  team  interviewed  several  staff  of  NMIC/ONI  23  Oetober  2007 
(Freeman  and  Hutehins,  2007).  The  interviews  were  struetured  to  elicit  comments  about  (1)  a 
draft  workflow  for  MDA  aetivities  surrounding  a  traeking  and  E-MIO  seenario  and  (2)  the  utility 
of  MDA  Spiral- 1  teehnologies  for  their  aetivities.  NPS  interviewed;  an  information  systems 
manager,  a  Wateh  Floor  COP  manager,  and  a  specialist  in  boarding  operations  and  data.  An 
informal  interview  was  conducted  with  the  head  of  the  Advaneed  Maritime  Analysis  Cell,  and 
with  the  lead  for  a  DoDAF  arehiteeture  effort  focused  on  the  intelligenee  day  shops.  All 
interviews  were  held  at  the  unelassified  level. 

Data  were  gathered  that  extended  the  MDA  workflow  model.  In  addition,  the  interviewees 
raised  several  issues  related  to  MDA  Spiral- 1  teehnologies: 

•  ONI  eontinuously  monitors  220-350  Vessels  of  Interest  (VOIs).  The  wateh  floor  -  staffed 
by  13  people  -  handles  as  many  as  six  formal  Requests  for  Information  (RFIs)  daily 
about  these  and  other  vessels,  15-20  informal  external  requests  daily,  and  a  small  number 
of  ONI  internal  queries.  The  watch  forwards  approximately  one  formal  RFI  to  analysis 
eells  (or  “day  shop”,  e.g.,  eounter-terrorism,  eounter-nareoties,  eounter-proliferation, 
homeland  defense)  eaeh  day.  ONI’s  eapability  to  handle  this  eurrent  volume  of  tasking  is 
hampered  by  difficulty  sharing  track  data  across  the  Navy,  insuffieient  training  resourees, 
insufficient  staffing  for  some  aetivities  (e.g.,  analysis  of  biometries  findings),  and  rapid 
turnover  of  staff  on  the  watch  floor.  NMIC/ONI  is  recruiting  several  hundred  additional 
staff  However,  staff  eapaeity  is  eurrently  a  eoneern. 

•  Several  Spiral- 1  teehnologies  are  seen  to  have  partieularly  high  value  within  ONI  or  as 
data  feeds  to  it:  CMA,  TRIPWIRE,  TAANDEM,  EASTCAP,  and  EMIO  wireless. 
However,  ONI  expressed  eoneern  that  (a)  the  interoperability  and  integration  of  these  and 
other  tools  (e.g.,  with  GCCS)  was  not  defined;  (b)  the  proeess  for  aeorediting  new  tools 
for  operational  use  is  long  (approximately  and  months)  and  somewhat  uncertain;  (e)  the 
tools  primarily  inerease  the  volume  of  data  available  for  analysis  but  do  not  help  analysts 
to  manage  those  data;  (d)  the  tools  do  not  strongly  enhanee  the  eapability  to  rapidly, 
reliably  prediet  aetivity  given  eyelieal  and  emergent  events,  or  infer  intent  or  eulpability 
from  seant  entity-relationship  data;  and  (e)  the  provision  of  tools  (e.g.,  Google  Apps  for 
eollaboration)  is  insuffieient  to  provide  the  intended  eapability  (e.g.,  improved 
eollaboration)  without  new  proeesses  and  training. 


83 


7.2.4  Interviews  at  NAVCENT 


Representatives  of  the  NPS  team  interviewed  several  staff  of  NAVCENT  11-15  November  2007 
(Freeman,  J.  and  MacKinnon,  D.,  2007).  The  interviews  were  structured  to  elicit  comments 
about  (1)  a  draft  workflow  for  MDA  activities  surrounding  a  tracking  and  E-MIO  scenario,  and 
(2)  the  utility  of  MDA  Spiral- 1  technologies  for  their  activities.  NPS  interviewed:  the  ONA 
Director  (N2),  the  Deputy  ONA  Director  (N2),  a  Communications  Information  Systems  officer 
(N6),  an  Information  Management  Officer  (N6IM),  the  Deputy  Director  of  Future  Plans,  ONI’s 
embedded  analyst  in  the  ONA,  an  ONA  MIO  specialist,  and  several  representatives  of  the  COPS, 
All  interviews  were  unclassified. 

Data  were  gathered  that  extended  the  MDA  workflow  model.  In  addition,  the  interviewees 
raised  several  issues  related  to  MDA  Spiral- 1  technologies: 

•  MDA  supports,  but  is  subordinate  to  the  primary  missions  of  NAVCENT:  maritime 
security,  anti-terror,  and  Iran.  The  prospect  of  receiving  Spiral- 1  technologies  sparked 
several  concerns:  the  relevance  of  the  technology  effort  to  primary  missions,  the  shortage 
of  personnel  and  high  rate  of  turnover  (10%  monthly),  concerns  about  training  staff  to 
use  technologies  effectively  for  NAVCENT  billets  and  processes,  concerns  about  system 
reliability  and  maintenance,  the  possibility  of  reduced  manning  as  a  result  of  MDA 
automation,  and  the  prospect  that  the  Flag  might  embark  from  NAVCENT.  These 
concerns  have  led  NAVCENT  leadership  to  consider  whether  many  MDA  activities  and 
Spiral- 1  technologies  should  be  housed  at  a  JIOC  or  at  ONI,  provided  that  those 
institutions  can  reliably  maintain  awareness  of  NAVCENT’s  mission  focus.  That  said, 
NAVCENT  leadership  views  positively  the  potential  Spiral-2  initiative  to  combine  the 
shore-based  radars  of  many  nations  with  AIS  data.  This  capability  would  benefit 
operations  in  the  MOC,  and  also  strengthen  partnerships  in  the  region. 

•  The  knowledge  of  the  Spiral- 1  technologies  among  NAVCENT  staff  (at  the  time  of  the 
interviews)  was  scant,  and  so  they  had  limited  ability  to  assess  the  utility  of  these 
technologies.  Watchfloor  personnel  see  value  in  technologies  that  triggers  or  alerts 
concerning  specific  tracks.  They  state  that  they  are  unlikely  to  use  technologies  that 
require  data  mining  or  fusion  across  multiple  sources. 


84 


8.0  Appendix  B:  Process  Engineering  &  Alignment 
Workshop  Results 


The  findings  of  field  interviews  results  were  validated  and  extended  in  two  workshops,  reported 
here:  a  Process  Engineering  Workshop  conducted  to  vet  the  MDA  workflow  and  a  Process 
Alignment  Workshop  conducted  to  map  the  MDA  workflow  to  the  MHQ  with  MOC  process 
architecture. 


8.1  Process  Engineering  Workshop 

The  Naval  Postgraduate  School  hosted  an  MDA  Process  Engineering  Workshop  (PEW)  hosted 
by  15-17  January  2008.  Representatives  of  the  following  organizations  participated  in  the  PEW: 
ASN  RDA,  C3E,  COTE,  Dept,  of  the  Under  Secretary  of  the  Navy,  DISA,  HEE  EEC,  JITIC, 
METRON,  MIECLANT,  MIECPAC,  NAVCENT,  NAVNETWARCOM,  NCIS,  NORTHCOM, 
NPS,  NRL,  NWDC,  ONI,  OPNAV,  PMW  120,  and  SPAWAR.  Also  participating  were  subject 
matter  experts  (SMEs)  from  several  of  the  MDA  Spiral- 1  technologies,  domain  experts  (‘gray 
beards’),  representatives  from  the  Trident  Warrior  2008  (TW08)  operational  experiment  where 
many  of  the  MDA  Spiral- 1  technologies  will  be  assessed,  and  members  of  the  assessment  team 
(NPS,  Aptima,  Pacific  Sciences  &  Engineering,  WBB  Inc.). 

Participants  in  the  Process  Engineering  Workshop  assessed  the  utility  of  Spiral- 1  technologies 
for  each  MDA  activity.  In  general,  PEW  participants  asserted  that  each  organizational  node  that 
had  access  to  any  Spiral- 1  technologies  would  use  all  of  those  technologies  in  most  of  its 
activities.  Thus,  the  activities  (below)  that  involve  ONI  and  ONA  make  heavy  use  of  Spiral- 1 
technologies  because  (1)  many  of  the  Spiral- 1  technologies  are  designed  to  support  intelligence 
analysis  and  (2)  many  of  these  technologies  will  be  inserted  at  ONI  and  ONA. 

Activities  conducted  by  COPS,  POPS,  the  MOC  Director,  and  BWC  are  not  expected  to  benefit 
from  many  of  the  technologies,  according  to  PEW  participants.  One  exception  is  the  task  “MOC 
Director:  Define  CDRs  Estimate  &  COA”,  a  task  in  which  the  MOC  director  may  draw  on  the 
Common  Intelligence  Picture  (CIP),  Common  Operational  Picture  (COP),  and  other  data  sources 
to  develop,  critique,  and  select  courses  of  action. 

We  note  that  the  assessment  of  Spiral- 1  utility  by  PEW  participants  conflicts  somewhat  with  the 
assessment  by  the  NAVCENT  MOC.  In  particular,  NAVCENT  anticipates  that  (1)  the  BWC 
would  use  PASTC2AP  and  SMS/JPSC2  to  execute  task  “Assess  Tactical  Asset  Availability”  and 
the  IWO  would  use  PASTC2AP  to  execute  task  “Issue  RET”  In  general,  NAVCENT  and  the 
PEW  agreed  in  their  assessment  that  ONA  would  use  a  variety  of  Spiral- 1  technologies  in  its 
intelligence  analyses.  NAVCENT  indicated  that  CMA,  MAGNET,  PASTC2AP,  Google  Earth, 
and  SMS/JPSC  would  be  particularly  useful  to  ONA.  These  differences  in  perspective  between 
NAVCENT  and  PEW  participants  are  indicated  with  a  *  in  the  table  below. 

Activities  executed  by  Elect  assets  make  almost  no  use  of  the  technologies  in  the  table  below, 
because  the  Elect  activities  do  not  require  most  of  the  analysis  functions  provided  by  these 
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technologies  or  because  Fleet  assets  are  not  expected  to  receive  them.  E-MIO  is  a  notable 
exception;  Fleet  assets  will  receive  E-MIO  and  will  benefit  from  it,  per  the  table,  below. 

Note  that  Table  17  does  not  include  mappings  of  technology  to  activities  for  early-stage, 
intelligence  generation  activities  (by  MARLO,  CIEC,  NCIS,  the  COCOM,  the  International 
Maritime  Bureau,  etc.)  nor  to  MOC-to-MOC  handoff  activities. 
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Table  17.  The  perceived  utility  of  Spiral-1  technologies  for  each  MDA, 


Entity 

CMA 

TAANDEM 

MAGNET 

FastC2AP 

Global 

Trader 

Tripwire 

E-MIO 

Wireless 

Google  Apps 
&Chat 

Google 

Earth 

SMS/ 

JPSC2 

LInX 

Austr  AIS 

ONI:  Intel 

X 

X 

X 

Tbd 

X 

X 

Tbd 

Tbd 

X 

Tbd 

ONA:  Nominate  potential  VOI 

X 

*x 

X 

X 

* 

* 

ONA:  Validate/(Re)Prioritize 
VOI 

X 

X 

* 

*x 

*X 

X 

* 

MOC  Director: 
Receive/Decide/Route  VOI 

X 

COPS:  Process  VOI 

FOPS:  Process  VOI 

BWC:  Assess  Tactical  Asset 
Availability 

* 

* 

MOC  Director:  Define  CDRs 
Estimate  &  COA 

X 

X 

X 

X 

X 

X 

X 

CNO/NOO:  Approve  COA 

X 

X 

X 

MOC:  Coordinate  MOC -to - 
MOC  Handoff 

X 

X 

X 

IWO:  Issue  RFI 

* 

ONI  issues  RFI  to  MOC 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

ONA:  Process  RFI  (Issue, 

Fulfill,  Assess  Fulfilled) 

X 

*x 

X 

X 

*X 

*x 

*x 

X 

* 

*x 

*X 

ONI:  Process  RFI  (Issue, 

Fulfill,  Assess  Fulfilled) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

NCIS,  CIFC,  MARLO, 
MIFCPAC,  NGA:  Process  RFI 

X 

X 

X 

X 

BWC:  Communicate  Mission 
Orders 

Fleet  Asset:  Plan  &  Direct 

VBSS  Mission 

Fleet  Asset:  ISR  Data 

Collection 

Fleet  Asset:  Take 
Biometrics/Boarding  Data 

X 

BFC:  Analyze  Biometrics 
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Entity 

CMA 

TAANDEM 

MAGNET 

FastC2AP 

Global 

Trader 

Tripwire 

E-MIO 

Wireless 

Google  Apps 
&Chat 

Google 

Earth 

SMS/ 

JPSC2 

LInX 

Austr  AIS 

ONI:  Analyze  biometric 
findings 

X 

X 

X 

NGIC/ONI:  Store  biometric 
report 

ONI:  Analyze  Boarding  Data 

X 

X 

X 

X 

X 

X 

X 

Fleet  Asset:  Receive  Boarding 
Data  Analysis 

X 

ONA:  Analyze  Findings 

X 

*x 

X 

X 

*X 

*x 

*X 

X 

* 

*X 

*X 

Coalition:  Execute  VBSS 

Mission 

COPS:  Monitor  VBSS 

COPS:  Recommend  Change 
Mission/Revision  of  CAT 

Level 

COPS:  Recommend  Mission 
Complete 

ONA:  Monitor  Vessel  of 

Interest  on  Watch  List 

X 

*x 

X 

X 

*X 

*x 

*X 

X 

X 

*x 

*X 

Note:  An  “X”  in  this  table  indieates  that  the  aetivity  would  benefit  from  the  Spiral- 1  teehnology  in  the  opinion  of 
PEW  partieipants.  A  indieates  that  the  assessment  by  NAVCENT  MOC  is  eontrary  to  the  PEW  assessment. 


Participants  in  the  PEW  also  offered  a  number  of  eoneerns  about  fielding  Spiral- 1  MDA 
teehnologies.  Many  of  these  eoneerns  are  typieal  for  a  technology  insertion  program,  and  thus 
they  represent  ehallenges  of  program  management  and  eustomer  expeetation  management. 

•  Customization  of  MOCs 

o  Organizational  structures  and  missions  (that  eompete  with  MDA)  vary  between 
COCOMS  and  MOCs.  MDA  TTPs  need  to  be  suffieiently  flexible  to 
aeeommodate  these  differenees.  Alternatively,  a  variety  of  TTPs  (e.g.,  for  small 
vs.  large  MOCs)  may  be  needed. 

•  Manning 

o  Current  Navy  guidanee  does  not  require  a  reduetion  in  manning  resulting  from 
implementation  of  Spiral-1  technologies.  NAVCENT  and  PACEET  have  stated 
that  they  will  require  additional  staff  to  operate  and  maintain  the  teehnologies. 

•  Technology  capability 
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o  Some  Spiral- 1  technologies  are  prototypes.  In  at  least  one  case,  the  technology 
SME  warns  that  these  technologies  may  not  be  sufficiently  robust  for  use  by 
operational  forces  (e.g.,  false  alarm  rates  may  be  too  high),  and  that  their  proper 
place  for  now  is  at  reachback  institutions  (such  as  NMIC/ONI)  that  have  the 
backup  capacity  to  overcome  these  potential  failures.  Other  stakeholders  have 
expressed  concern  about  specific  technologies:  CMA  (number  of  databases 
delivered  vs.  number  of  databases  promised),  TAANDEM  (accreditation 
challenges),  EASTC2AP  (maturity  of  the  alerting  capability). 

Training 

o  NAVCENT  and  PACEET  have  expressed  concern  that  training  products  be 
delivered  with  the  systems,  and  that  this  training  address  organization-specific 
applications  of  the  technology. 

Technology  Installation 

o  Standardization:  The  unique  IS  environments  across  the  fleet  will  present  a 
challenge  to  technology  installers. 

o  Physical  capacity:  Some  sites  do  not  have  the  physical  space  to  accommodate 
additional  technologies,  particularly  if  each  technology  is  delivered  on  a  separate 
server.  NORTHCOM  is  a  case  in  point.  It  can  expand  its  IS  spaces  for  new 
servers  only  by  blasting  additional  rooms  into  the  mountain. 

o  Power  capacity:  The  old  infrastructure  at  some  sites  constrains  insertion.  ONI,  for 
examples,  requires  additional  electrical  power  for  every  significant  technology 
insertion.  Delivery  of  additional  power  can  take  half  a  year  or  more. 

Testing 

o  Metrics  are  needed  to  assess  the  effects  of  technology  insertion  relative  to  the 
current  state.  Unfortunately,  there  are  few  if  any  published  standards  that  define 
the  effectiveness  of  current  solutions  in  operations.  (Standards  for  the  Navy  Task 
Eist  pertain  to  training,  not  operational  use,  for  example). 

o  A  sufficiently  detailed  scenario  is  needed  to  drive  testing.  This  scenario  must 
systematically  address  the  variety  of  MDA  data  types  (vessel,  people,  cargo,  etc.), 
reporting  products,  node  interactions,  and  time  course  of  activity  in  a  problem  that 
involves  discovery,  analysis,  and  prosecution  of  VOIs.  Particularly  important 
challenges  in  MDA  are:  ISR  management,  collection  planning,  decisions 
regarding  opposed  and  unopposed  boardings,  and  tracking  neutrals.  In  addition, 
scenario  designers  should  consider  events  in  which  multiple  vessels  collaborate  in 
a  threat  incident,  either  through  cargo  transfer  between  vessels  or  by  synchronized 
tactical  actions  of  two  or  more  vessels.  TWOS  is  developing  a  scenario  using 
systematic  methods. 

Process  Analysis 

o  Additional  detail  is  needed  concerning  intelligence  analysis  processes  (monitor, 
collect,  fuse,  analyze,  and  disseminate).  This  analysis  is  being  conducted 
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independently  by  ONI,  but  that  proeess  has  only  reeently  begun  (e.g.,  analysis  of 
one  day  shop  was  eompleted  as  of  November  2007)  and  so  the  results  may  not  be 
available  to  support  Spiral- 1  testing. 

o  The  MDA  workflow  should  be  aligned  with  the  MHQ  w/MOC  proeess 

arehiteeture.  This  was  sueeessfully  addressed  in  a  Proeess  Alignment  Workshop 
29  January  2008. 


8.2  Process  Alignment  Workshop  Results 

A  Proeess  Alignment  Workshop  was  held  to  align  the  MDA  task  with  MHQ  w/MOC  proeesses. 
The  partieipants  in  the  workshop  were  approximately  20  warfighting  funetional  area  leads, 
system  and  proeess  arehiteets,  proeess  SMEs,  and  interoperability  experts.  Tim  Sorber  of  Klett 
Consulting  and  the  Seeond  Fleet  MHQ  w/MOC  arehiteeture  team  organized  the  session  and  eo- 
led  the  workshop  with  Dr.  Jared  Freeman  (Aptima)  for  the  NPS  team,  and  Greg  Allen  of  WBB. 

Partieipants  mapped  (1)  OV6C  diagrams  of  MDA  aetivities  to  (2)  the  MHQ  with  MOC  proeess 
arehiteeture  “SmartPaek.”  The  OV6C  was  developed  by  NAVNETWARCOM  (by  eontraetors 
WBB  and  Booz  Allen  Hamilton)  based  on  MDA  workflow  diagrams  developed  by  NPS  (see 
seetion  4.4). 

The  workshop  partieipants  eoneluded  that  virtually  all  MDA  aetivities  mapped  to  MHQ  with 
MOC  proeesses,  though  some  MDA  tasks  speeifie  to  handoff  between  MOCs  had  yet  to  be 
addressed  in  the  MHQ  with  MOC  arehiteeture  at  that  time.  Several  other  minor 
reeommendations  were  made  to  revise  the  doeuments.  These  are  speeified  in  Teehnieal  Report 
NPS-IS-08-002,  available  from  NPS. 


8.3  Recommendations 


On  the  basis  of  the  PEW  findings  and  the  field  analysis  of  MDA  workflow  (above)  the  NPS 
assessment  team  made  more  than  a  dozen  reeommendations  eoneerning  Spiral- 1  MDA 
teehnology,  its  fielding,  and  its  assessment. 


(1)  Pay  partieular  attention  to  planning  and  soeializing  the  plan  to  train  and  maintain 
support  Spiral- 1  teehnologies.  This  may  allay  eoneerns  in  CINPACFET, 
NAVCENT,  and  elsewhere  that  the  teehnology  delivery  is  not  paying  suffieient 
attention  to  the  human  faetor. 

(2)  Training  for  Spiral- 1  teehnology  users  and  maintainers  should  be  rapid, 
demonstrably  effeetive,  and  eustomized  to  loeal  missions  and  proeedures.  This 
will  address  eoneerns  that  teehnology  won’t  support  loeal  missions  and  proeesses 
for  staffs  with  high  turnover 
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(3)  Given  the  potential  that  staff  will  use  new  MDA  teehnologies  infrequently  in 
some  organizations,  they  may  forget  how  to  use  the  teehnologies  effieiently  and 
well.  Usability,  training,  and  teehnieal  support  will  be  partieularly  important 
predietors  of  sueeess  in  these  organizations.  These  should  be  a  foeus  of 
assessment 

(4)  Teehnology  providers  should  eontinue  to  brief  user  eommunities  eoneerning  the 
maturity  and  delivery  sehedules  for  teehnologies.  This  may  help  users  to  prepare 
for  the  speeifie  eapability  they  will  reeeive 

(5)  The  aeereditation  proeess  must  be  earefully  managed  aeross  the  Spiral- 1 
teehnologies.  Lessons  learned  should  be  used  to  aeeelerate  that  proeess. 

(6)  Interoperability  issues  should  be  assessed  and  managed  earefully.  Interoperability 
reduees  training  requirements,  faeilities  requirements,  and  eognitive  load 
(imposed  when  users  must  remember  results  from  one  system  while  they  use 
another,  or  fuse  information  mentally). 

(7)  Teehnology  roll-outs  to  NAVCENT  should  be  (1)  seleetive,  to  respeet 
NAVCENT’s  vision  of  its  role  and  eapabilities,  and  (2)  strongly  supported  with 
training  and  teehnieal  aid  to  ensure  that  NAVCENT  evaluates  those  teehnologies 
in  the  most  positive  light. 

(8)  In  deliveries  to  NAVCENT,  emphasis  should  be  plaeed  on  teehnologies  that 
support  alerting  over  teehnologies  that  support  data  mining  and  fusion,  for  whieh 
NAVCENT  is  not  eurrently  well  staffed. 

(9)  Teehnology  roll-outs  to  MIECPAC  should  be  (1)  seleetive  to  support  the 
organization’s  missions,  and  (2)  inelude  support  to  eustomize  the  teehnology  for 
regulatory  enforeement  missions. 

(10)  Teehnologies  with  low  aeouraey  or  reliability  should  be  plaeed  in  reaehbaek 
eenters  rather  than  front  line  operational  eenters. 

(11)  Spiral- 1  fielding  will  require  attention  to  marked  differenees  between 
installation  environments  with  respeet  to  physieal  spaee,  power  eapaeity,  ete. 

(12)  Teehnology  assessments  should  foeus  largely  on  the  effeets  of  Spiral- 1 
insertion  on  (1)  aeeess  to  information  that  was  previously  inaoeessible  or 
diffieult  to  aeeess,  (2)  speed  of  deeision  making,  and  (3)  aeouraey  of  deoision 
making.  Measures  on  these  aotivities  will  be  of  great  interest  to  user 
eommunities. 

(13)  The  MDA  assessment  should  aoourately  measure  the  impaot  of  new 
teehnologies  on  training  and  maintenanoe  requirements  for  Spiral- 1 
teehnologies  relative  to  ourrent  requirements.  This  will  help  user  eommunities 
prediot  and  manage  these  oosts. 

(14)  MDA  “to  be”  proeess  must  be  standardized  to  ensure  the  interoperability  of 
MDA  stakeholders,  but  oustomized  to  looal  missions  and  eapabilities.  This  is  a 
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significant  challenge.  It  is  one  faced  by  the  MHQ  w/MOC  proeess  arehitecture 
team,  and  their  strategies  should  be  studied  and  applied  here. 

(15)  A  “to  be”  workflow  for  MDA  at  ONI  using  Spiral- 1  teehnologies  should  be 
developed  in  eollaboration  with  the  team  that  is  eondueting  DoDAF  modeling 
of  ONI  analysis  proeesses. 

In  addition,  a  review  of  the  MDA  workflow  (not  reported  here)  indieated  the  potential  for 
inereased  volume  of  information  flow  from  intelligenee  entities  to  operational  entities.  The 
following  reeommendation  was  made: 

(16)  Measure  and  manage  the  rate  of  information  flow  from  intelligenee  entities  to 
the  MOC  to  ensure  that  the  MOC  is  able  to  aecurately  filter,  prioritize,  and 
proeess  ineoming  intelligenee. 


92 


8.4  Workflow  Architectures 


These  OV-6C  diagrams  of  MDA  process  were  developed  by  NAVNETWARCOM  contractors 
WBB  and  Booz  Allen  Hamilton  from  workflow  data  and  diagrams  developed  by  NFS. 


Figure  1.  MDA  OV6C:  Top  Level  Process. 
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Figure  2.  MDA  0V6C:  Coordinate  Handoff 
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10.  Process  RFI  (NMC/CNI)_v0.2J7  Jan  08  (Business  Process) 

System  Architect 

WMlJan23.2008  12:43 

Itis  dla^am  describes  the  (taft  ONI  process  for  handling  RFIs  as  provided  In  notes  form  APTIMA 
based  on  discussions  between  Jared  Freeman  and  ONI  representatives  Jim  Stallings,  LTLange  and 

Paul  Oarroll.  Hwas  also  reviewed  by  ONI  repLTKingatthe  Process  EngIneeringWorkshcponI? 

JAN  at  NFS  Uanterey.  No  reqjredchanges  were  Identified. 

Figure  3.  MDA  0V6C:  Process  RFI 


Figure  4.  MDA  0V6C:  Provide  Info/Intel 
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8.5  MDA  Workflow 


The  workflow  documents  presented  below  define  workflow  within  an  organization  or,  when  that 
organization  has  many  activities,  within  a  cell  or  role.  The  workflow  incorporates  data  developed 
by  NFS  in  interviews  and  workshops,  as  well  as  inferences  made  by  NAVNETWARCOM 
contractor  WBB  (and  verified  in  the  workshops  above).  These  data  are  represented  in  the  MDA 
OV-6C  data  developed  by  NAVNETWARCOM.  The  graphs,  below,  were  generated  from  a 
current  version  of  that  data  set  delivered  by  Booz  Allen  Hamilton  to  Aptima  on  29  February 
2008. 

The  following  conventions  are  used  here; 

•  The  activities  of  a  specific  organization,  cell,  or  role  are  framed  in  a  rectangle.  Their 
interactions  with  other  entities  are  indicated  by  arcs  to  nodes  outside  that  rectangle. 

•  Each  activity  node  specifies: 

o  The  activity  (e.g..  Prepare  HR)  or  information  product  in  the  Collaborative 
Information  Environment  (CIE) 

o  The  entity  who  performs  the  activity  (e.g.,  NCIS) 

o  In  some  cases,  the  corresponding  MHQ  w/MOC  Core  Process.  (The  mapping  of 
MDA  tasks  to  Core  Processes  was  conducted  for  high-level  tasks  only.  Tasks  that 
are  not  marked  with  Core  Processes  here  are  subordinate  to  other  tasks  that  are  so 
marked. 

o  The  node  identifiers  from  DoDAF  data. 

•  Where  multiple  arcs  emerge  from  a  node,  these  sometimes  represent  options  (“or”,  not 
“and”). 

The  reader  can  view  these  detailed  images  by  expanding  them  (select  and  drag  the  comers)  and 
zooming  the  view.  These  images  are  jpeg  files  that  should  port  well  to  other  applications  for 
better  viewing. 

The  data,  scripts,  and  application  used  to  generate  these  graphs  are  available  upon  request. 

Figure  5  depicts  all  MDA  workflow  identified  in  this  project.  Section  15.0  presents  workflow 
diagrams  for  those  units  involved  in  MDA  operations. 
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9.0  Appendix  C:  VBSS  School  Results 


NPS  officer  students  attended  the  Visit  Board  Search  and  Seizure  (VBSS)  Sehool  for  training 
and  to  gather  assessment  information  on  the  Tactical  EMIO  System  (TES).  The  following  is  the 
report  of  LT  Dalton  Clarke. 

On  June  19,  2008  the  second  phase  testing  of  the  Tactical  EMIO  System  (TES)  was  conducted  at  the  VBSS  School 
in  San  Diego.  Testing  planned  for  the  evaluation  and  integration  of  5  Tactical  EMIO  Devices  (TED),  I  Tactical 
EMIO  Maritime  PC  (TEMP),  and  Maritime  BGAN  EMIO  Terminal  (MBET).  Figure  1  provides  a  TES  Overview 
and  illustrates  the  workflow  process  between  the  TED,  TEMP,  and  MBET. 


TES  Overview 


DFTiwI  prmiids  an  IMaflaea  CanM 
Dociment  msl  deftnct  tw  data  Viat  ina  teMP 
tanoi  13  Ita  Server 


Eigure  6.  Digital  Eorce  Technology's  (DFT)  TES  Overview 


9.1  Training 

Students  10  (in  total)  attending  the  VBSS  School  received  45  minutes  of  lecture  and  hands-on  training  in  the  use  of 
the  TED,  TEMP,  and  MBET  prior  to  fielding  for  experimentation  on  the  USS  RUSHMORE  (LSD  48).  The 
facilitator  (civilian  contractor)  introduced  all  three  EMIO  systems  while  placing  particular  emphasis  on  the 
durability  and  user  friendliness  of  the  Graphic  User  Interface  (GUI).  TEDs  were  then  given  to  the  students  to  link  to 
the  TEMP  via  the  wireless  network  (802.11).  However,  the  TEMP  had  minor  difficulties  linking  to  the  TED  due  to 
IP  address  issues.  Students  were  able  to  configure  their  TEDs  for  data  capturing  objectives  once  802.11  links  were 
established  between  the  TEMP  and  TED.  Due  to  time  constraints,  students  could  not  link  units  to  the  MBET  system 
for  data  transfer.  Baseline  training  surveys  were  given  to  all  10  students  which  indicated  that  they  had  a  basic 
understanding  about  the  operation  of  the  EMIO  units. 
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9.2  Experimentation 

Testing  commenced  at  16:46.  Students  boarded  the  target  vessel  USS  RUSHMORE  via  small  boats  and  hoisted  up 
all  of  EMIO  units  incased  in  a  bag.  Contractors  assisted  students  using  2  of  5  TEDs  due  to  limited  training 
conducted  on  TED  units.  Data  capturing  commenced  once  the  vessel  was  secured  by  VBSS  teams.  The  Boarding 
Officer  assigned  specific  data  objectives  through  the  TEMP  device  for  the  TED  devices  to  complete.  Individual 
data  was  collected  from  crew  members  located  in  DC  Central  and  Officer  Country.  Once  all  data  objectives  were 
completed,  TED  devices  were  brought  to  the  pilot  house  to  transfer  data  to  the  TEMP  device.  Although  wireless, 
the  TED  devices  were  required  to  be  in  the  vicinity  of  the  TEMP  in  order  to  download  data  captured  during  the 
boarding.  The  RF  signals  were  not  strong  enough  to  transmit  data  when  team  members  were  below  decks.  The 
MBET  device,  in  turn,  failed  to  transfer  data  due  to  environmental  issues  and  weak  RF  signal  range  of  commercial 
satellite.  Contractors  eventually  departed  the  target  vessel  and  drove  inland  with  the  TEMP  and  MBET  device  to 
acquire  a  stronger  signal.  The  MBET  link  was  then  acquired  and  successfully  transmitted  data  from  the  TEMP 
device.  Neither  the  Boarding  Officer  nor  the  students  witnessed  the  data  transfer.  Contractors  asserted  that  the 
satellite  connectivity  will  not  be  a  concern  in  the  current  AOR.  An  EMIO  general  user  survey  was  given  to  VBSS 
members  who  operated  TED  and  TEMP  devices.  Surveys  evaluating  the  practical  operation  of  the  MBET  device 
were  not  handed  out  due  to  students  having  little  interaction  with  the  device. 

9.3  Observations 

With  regard  to  training,  students  were  not  granted  enough  time  to  practice  operating  all  three  systems  (TED,  TEMP, 
and  MBET).  Students  asked  many  questions  during  the  training  which  provided  useful  feedback  for  the  contractors 
to  learn  the  perspective  of  the  user.  Some  questions  focused  on  why  the  units  did  not  incorporate  biometrics  and 
camera  capabilities  into  one  unit.  The  contractor  explained  that  biometrics  were  not  included  due  to  privacy  issues 
and  that  these  units,  thus  far,  were  not  being  used  by  coalition  forces.  He  anticipated  that  the  camera  will  be 
incorporated  in  the  next  version  of  EMIO  system. 

Observing  the  practical  operation  of  the  TED  in  the  field,  indicated  that  the  students  would  require 
additional  training  to  fully  operate  the  TED.  Yet,  the  students  were  able  to  use  the  features  effectively  to  capture 
required  data.  Additionally,  mobility  of  the  TED  was  a  clear  advantage  allowing  for  the  collection  of  data  from 
multiple  locations  within  the  vessel.  One  limitation  noted  was  that  the  TED  must  be  within  the  vicinity  of  the 
TEMP  to  download  the  data  captured.  The  Boarding  Officer  was  not  able  to  demonstrate  the  transfer  of  data  via  the 
MBET  due  to  the  satellite  connectivity  concerns  mentioned  above. 

Observations  of  the  MBET  demonstrated  that: 

•  Its  design  was  very  rugged  and  seemed  to  prevent  water  intrusion. 

•  A  heat  sink  was  also  attached  to  dissipate  heat. 

•  The  support  handle  for  the  MBET  will  eventually  require  a  different  material  to  prevent  salt  water 

corrosion. 

•  The  MBET  device  was  not  light  enough  to  be  carried  on  a  VBSS  person. 

Students  indicated  via  user  surveys  that: 

•  The  equipment  should  integrate  the  camera  along  with  biometric  capability 

•  The  TEDs  had  too  many  functions  to  organize  the  data. 

•  They  would  prefer  less  required  steps  to  configure  the  system 

•  The  EMIO  gear  was  very  mobile 

•  The  graphic  user  interface  was  intuitive  and  simple  to  use. 

The  survey  for  the  MBET  was  not  filled  out  due  to  the  lack  of  training  and  satellite  connectivity  concerns. 
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10.0  Appendix  D:  FAIRGAME  QRA  Results 


The  following  is  the  QRA  results  message,  modified  to  ehange  from  message  format  to  one 
easily  read.  The  eontent  has  not  been  changed. 

10.1  Purpose 

This  is  a  report  of  COMOPTEVFOR's  Quick  Reaction  Assessment  (QRA)  of  the  Maritime 
Domain  Awareness  (MDA)  Spiral- 1  prototype  leave-behind  MDA  capability.  Per  Refs  A  and  B, 
the  purpose  of  the  QRA  was  to  determine  potential  operational  effectiveness,  potential 
operational  suitability,  and  considerations  for  deploying  leave-behind  technologies  at  identified 
nodes.  Additionally,  the  adequacy  of  training,  documentation,  and  the  ability  to  locate,  analyze, 
and  pass  target  information  to  other  nodes,  and  receive  analytically  derived  information  from 
queries  to  National  Maritime  Intelligence  Center  (NMIC)  were  to  be  assessed. 

Per  Ref  D,  the  QRA  was  conducted  from  16  Jun-23  Jul  2008.  Per  Ref  E,  user  feedback  was 
solicited  and  provided  a  majority  of  this  report’s  findings. 


10.2  Summary 

MDA  Spiral- 1  is  potentially  operationally  effective  and  potentially  operationally  suitable. 
Leave-behind  technologies  will  enhance  the  warfighter’s  ability  in  maintaining  situational 
awareness  of  the  global  maritime  domain. 

•  MDA  related  training  and  documentation  provide  the  basic  building  blocks  to  employ  the 
MDA  suite  but  require  considerable  modification  and  review. 

•  MDA  Sprial-1  provided  useful  information  on  vessel  location,  cargo,  and  people. 

•  However,  the  ability  to  conduct  analysis  was  limited  by  incomplete  data  at  each  node. 

•  The  ability  to  collaborate  between  nodes  was  affected  by  the  lack  of  commonality  of  data 
at  the  nodes. 

•  Users  did  not  have  information  on  how  or  what  global  data  was  provided  to  the  nodes  and 
had  difficulty  resolving  discrepancies  and  constructing  track  histories. 

10.3  Scope  of  Assessment 

The  QRA  assessed  the  following  Spiral- 1  technologies:  Comprehensive  Maritime  Awareness 
(CMA)  Joint  Capability  Technology  Demonstration  (JCTD) 

•  Law  Enforcement  Information  Exchange  (LINX) 

•  Cargo  Link 

•  Maritime  Awareness  Global  Network  (MAGNET) 

•  Fast  Connectivity  for  Coalition  and  Agents  Project  (FASTC2AP) 

•  TRIPWIRE  Analytic  Capability  (TAC) 

•  Google  Earth  Fusion  Services 
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•  Tactical  Expanded  Maritime  Interception  Operations  System  (TES) 

•  MDA  Data  Sharing  Community  of  Interest  (MDA  DS  COI), 

•  SEAPORT  Collaboration  at  Sea  (SEAPORT  CAS), 

•  Sensor  Management  Systems/Joint  Perimeter  Surveillanee  Command  and  Control 
(SMS/JPSC2). 

The  teehnologies  were  not  integrated  to  act  as  a  system-of-systems  capability,  but  rather  as 
independent  eapabilities,  with  no  two  nodes  having  the  same  Spiral- 1  capabilities,  set-up,  or 
MDA  focus  area. 


10.4  Project  Testing 

The  assessment  was  eonducted  at  three  venues:  Trident  Warrior  08,  Navy  Visit  Board  Seareh 
and  Seizure  (VBSS)  School,  and  SPAWAR-Sponsored  Exercise  EAIRGAME  08-2. 

A.  Trident  Warrior  08.  Trident  Warrior  is  an  annual  experiment  which  examines  new 
teehnologies  and  proeesses  among  operational  fleet  users.  Trident  Warrior  08  hosted  a  subset  of 
the  Spiral- 1  suite  (EINX,  CMA,  CARGO  EINK,  and  MDA  DS  COI)  at  COMPACELT, 

Maritime  Intelligence  Eusion  Center  Paeific  (MIECPAC),  C3P,  MIECEANT,  NMIC, 

and  various  NCIS  field  sites. 

B.  The  Navy  VBSS  school  in  San  Diego  hosted  the  TES  demonstration. 

C.  EAIRGAME  08-2.  EAIRGAME  08-2  was  a  SPAWAR  sponsored  event  that  faeilitated  the 
demonstration  of  the  Spiral- 1  suite  capabilities.  EAIRGAME  hosted  the  entire  Spiral- 1  suite 
with  the  exeeption  of  EINX  and  operational  testing  of  the  RES  at  COMUSNAVCENT.  The  sites 
represented  were  COMUSNAVCENT,  COMPACEET,  MIECPAC,  MIECEANT,  and  NMIC. 

The  SPAWAR  T&E  team  assisted  by  the  SSC  San  Diego  Advaneed  Concept  Engineering 
Eaboratory  (ACE  lab)  acted  as  White  Cell. 

D.  MDA  MISSION  AREAS.  The  MDA  mission  are  differed  according  to  COCOM  and  Area  of 
Responsibility  (AOR).  Because  of  the  mission  area  differenees,  user  feedback  varied  by  node. 


10.5  Limitations 

A.  The  TES  system  was  demonstrated  using  U.S.  NAVY  VBSS  operators  vice  intended 
coalition  users. 

B.  The  time  period  between  teehnology  installation  and  the  QRA  was  not  sufficient  for  the  user 
to  become  proficient  at  using  the  new  tools.  Some  user  feedbaek  is  based  on  little  or  no  training. 
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C.  U.S.  NAVY  users  did  not  have  direet  aeeess  to  the  MAGET  system.  MAGNET  data  was 
only  aeeessible  through  CMA  and  was  not  tested  as  a  stand-alone  capability. 


D.  The  AIS  system  was  accessible  only  through  CMA,  MDA  DS  COI,  and  Google  Earth  and 
was  not  tested  as  a  stand-alone  capability. 

E.  The  Global  Trader  system  was  accessible  only  via  Cargo  Eink  web  access  and  was  not  tested 
as  a  system. 

E.  Eimited  capability  to  extract  data  from  MDA  Spiral-I  systems  and  limited  time  to  analyze  the 
extracted  data  limited  the  QRA  primarily  to  qualitative  methods.  Because  of  time  limitations, 
quantitative  measures  were  scored  on  only  a  small  sample  of  the  extracted  data. 

G.  Real-world  data  was  used  as  ground  truth  for  the  EAIRGAME  scenarios  and,  as  such,  some 
capabilities  may  not  have  been  fully  observed  due  to  exercise  limitations. 

H.  The  San  Diego  SMS/JPSC2  system  was  not  observed  as  a  stand-alone  system.  However, 
SMS/JPSC2  data  were  observed  via  the  MDA  DS  COI. 


10.6  Observations 

A.  The  critical  operational  issue  assessment  was  of  Spiral- 1  technologies  functionality  as 
operated  during  this  assessment. 

White  =  not  assessed/tested 
Green  =  sufficient  functionality 
Yellow  =  partial  functionality 
Red  =  limited  functionality 

EEEECTIVENESS  CRITICAL  OPERATIONAL  ISSUES 


E-I 

Monitor 

Y  ellow 

E-2 

Collect 

Yellow 

E-3 

Euse 

Green 

E-4 

Analyze 

Red 

E-5 

Disseminate/Collaborate 

Yellow 

E-6 

Operating  Picture 

Y  ellow 

E-7 

Communications  Capability 

Green 

SUITABILITY  CRITICAL  OPERATIONAL  ISSUES 


S-1  Reliability 

Green 

S-2  Maintainability 

White 

S-3  Availability 

Green 

S-4  Logistic  Supportability 

Green 
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S-5 

Compatibility 

Green 

S-6 

Interoperability 

Green 

S-7 

Training 

Yellow 

S-8 

Human  Factors 

Green 

S-9 

Safety 

Green 

S-10  Documentation 

Yellow 

B.  EFFECTIVENESS  CRITICAL  OPERATIONAL  ISSUES 

(I)  E-1  Monitor.  Will  MDA  Spiral- 1  technologies  enhance  the  warfighters  ability  to  monitor  the 
global  maritime  domain?  (YELLOW) 

•  CMA  provided  searchable  data  on  vessel  location,  cargo,  and  people  but  there  were  gaps 
in  track  data,  a  lack  of  commonality  of  data  between  nodes,  and  insufficient  information 
on  data  sources. 

•  The  user’s  ability  to  monitor  vessel,  person,  and  cargo  data  was  severely  degraded  by 
gaps  in  track  data  coverage.  When  a  node’s  CMA  server  was  down,  or  data  was  not 
transmitted,  the  data  not  received  was  not  recoverable.  The  Naval  Research  Laboratory 
(NRL)  limited  the  National  Technical  Means  (NTM)  data  source  input  to  CMA  to  14 
hours  per  day  and  filtered  the  data  that  was  provided.  This  resulted  in  a  gap  of  data 
which  had  a  negative  impact  across  all  AORs. 

•  On  several  occasions  a  query  of  current  vessel  position  returned  multiple  vessel 
positions,  each  with  different  metadata  (time,  cargo,  flag)  attached. 

•  Users  noted  a  general  inconsistency  of  the  CMA  vessel  database  between  nodes  within 
the  same  AOR.  An  analysis  was  performed  on  the  commonality/consistency  of  track  data 
held  by  the  NAVCENT  and  PACFLT  CMA  systems  within  a  5  by  10  deg  geographic 
region  in  the  Indian  Ocean.  Although  97%  of  the  tracks  were  held  by  both  systems, 
significant  track  metadata  differences  were  observed  in  15%  of  the  tracks,  and  track 
history  data  were  only  51%  common  (77%  common  for  tracks  going  from  the 
NAVCENT  AOR  to  the  PACFLT  AOR,  and  29%  common  for  tracks  going  in  the  other 
direction).  Some  track  history  data  in  one  site’s  CMA  were  not  in  the  other  site’s 
CMA. 

•  Users  consistently  cross  referenced  other  data  sources  to  validate  query  responses 
returned  by  CMA.  The  user  had  limited  information  on  information  feeds  into  CMA. 

The  user’s  uncertainty  in  the  completeness  of  the  data  affected  confidence  in  the  search 
results. 

•  The  percentage  of  worldwide  cargo  data  fed  to  CMA  by  Global  Trader  was  not  known. 
The  percentage  of  worldwide  cargo  data  available  via  Cargo  Link  was  not  known. 

•  The  ability  to  monitor  un-equated  vessels  using  only  FLINT  data  was  not  observed. 
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(2)  E-2  Collect.  Will  MDA  Spiral- 1  technologies  enhance  the  collection  capability  of  the  global 
maritime  domain?  (YELLOW) 

•  The  lack  of  commonality  between  nodes  made  the  aggregation  of  information  difficult. 
MDA  Spiral- 1  did  not  contain  a  central  repository  of  worldwide  vessel  movement  data. 
Differences  in  node  metadata  required  additional  research  on  the  part  of  the  user, 
increasing  the  time  required  to  perform  a  mission. 

•  The  user  had  no  knowledge  of  foreign-to-foreign  cargo  sharing  agreements  or  the 
amount  of  cargo  data  actually  available,  which  affected  the  ability  to  build  and  evaluate 
cargo  queries. 

•  The  TES  System  Operational  Verification  Testing  (SOVT)  indicated  a  successful  file 
upload  to  the  SEAPORT  CAS.  However,  the  afloat  capability  was  not  observed. 

(3)  E-2  Fuse.  Will  MDA  Spiral- 1  increase  the  ability  to  fuse  data  from  information  gleaned 
across  COCOM  AORs,  interagency  domains,  and  across  multiple  security  classification 
domains?  (GREEN) 

•  Spiral- 1  tools  increased  the  user’s  ability  to  fuse  vessel,  people,  and  cargo  data  from 
interagency  sources  but  provided  limited  capability  to  fuse  data  across  AORS  and 
security  classification  domains.  The  user  had  a  limited  ability  to  view  a  composite  track 
with  all  gathered  information.  The  tools  did  not  allow  for  user-defined  pedigree  rule  sets. 

•  Users  sometimes  lost  track  of  the  original  security  classification  of  the  information  they 
wished  to  disseminate.  This  increased  the  likelihood  of  a  security  violation  as  a  result  of 
passing  classified  information  on  the  wrong  domain. 

(4)  E-4  Analyze.  Will  MDA  Spiral- 1  enhance  the  warfighter’s  ability  to  accurately  and  timely 
analyze  the  global  maritime  domain  picture?  (RED) 

•  The  ability  to  capture  and  store  baseline/normal  maritime  movement  patterns  was  not 
observed.  Spiral- 1  tools  did  not  alert  users  to  deviations  from  normal  route  or  behavior 
patterns. 

•  EASTC2AP  accessed  historical  track  data  but  did  not  capture  and  store  baseline 
movement  patterns.  PASTC2AP  provided  basic  anomalous  detection  capability  based 
upon  a  user-defined  agent  using  historical  track,  not  baseline/normal  behavior  patterns. 

•  CMA  track  data  was  primarily  limited  to  tracks  within  the  AOR.  Lack  of  track  data 
outside  the  AOR  limited  the  user’s  ability  to  analyze  maritime  movement  patterns.  Baps 
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in  track  and  metadata  also  affected  analysis  of  data  within  a  node  and  affeeted  the  ability 
to  collaborate  between  nodes. 

•  The  MDA  DS  COI  may  provide  a  basie  anomalous  detection  capability  but  since  the 
users  were  not  trained  on  the  full  eapability  it  was  not  observed. 

•  Cargo  Link  did  not  provide  a  eargo  anomaly  deteetion  capability. 

• 

•  Spiral- 1  did  not  provide  new  toolsets  to  enhance  collaboration  between  nodes.  Spiral- 1 
teehnologies  did  allow  for  the  easy  export  of  seareh  response  data  for  use  with  existing 
eollaborative  tools.  The  collaboration  tool  of  choice  was  MS  Chat.  NCIS  used  the  web- 
based  Foree  Protection  Portal  for  cross  AOR  collaboration.  The  portal  acted  as  a  eentral 
repository  for  all  information  eoneeming  the  investigative  scenarios. 

•  Spiral- 1  tools  did  not  provide  additional  capability  to  establish  baseline  normal  eivil 
maritime  operations  worldwide  and  threat  assessment  eriteria.  CMA  and  FASTC2AP 
could  alert  based  upon  a  geographic  point/area/proximity,  but  did  not  support  alerts 
employing  algorithms  based  upon  baseline  maritime  operations. 

•  Spiral- 1  tools  did  not  conduct  statistical  analysis  of  data  gaps  in  order  to  identify  potential 
new  sources  of  information  and  drive  new  colleetions. 

(5)  E-5.  Disseminate/CoUaborate.  Will  MDA  Spiral- 1  provide  timely  and  accurate  correlation, 
storage,  and  display  of  operational  data  and  other  tactically  significant  information  in  a  manner 
that  will  support  the  full  spectrum  of  global  information  needs?  (YELLOW) 

•  Spiral- 1  did  not  automatieally  establish  or  display  threat  assignments  based  upon  a  user- 
defined  alert. 

•  EASTC2AP  alerted  the  user,  per  user-defined  alerts,  when  a  vessel  meets  the  alert 
thresholds.  The  user  then  conducted  a  threat  analysis  on  the  vessel.  The  TAC  system 
will  alert  the  user  that  there  is  a  data  element  that  meets  seareh  criteria. 

•  MDA  Spiral- 1  was  not  a  composite  system  and  the  ability  to  aggregate  and  replicate 
MDA  data  on  a  global  seale  was  not  possible.  CMA  did  not  replieate  its  database 
globally,  eaeh  node  maintained  its  own  loeal  database. 

•  MDA  DS  COI  was  observed  at  MIECPAC  and  MIECLANT  only,  therefore  the  global 
effeetiveness  and  suitability  of  the  technology  cannot  be  assessed. 

•  The  NAVCENT  TAC  system  and  Cargo  Eink  were  the  only  Spiral- 1  teehnologies  that 
had  direct  access  to  ONI  historical  maritime  data.  The  SEAPORT  CAS  use  was 
observed  at  NAVCENT  only. 
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(6)  E-6  Operating  Picture.  Will  MDA  Spiral- 1  enhance  the  user ’s  ability  to  maintain  an 

operational  picture  that  supports  access  to  tactical,  operational,  and  strategic  data  elements 
across  multiple  A  ORs  ?  (YELLOW) 

•  Based  on  analysis  of  4  hours  of  extracted  data  in  a  littoral  area, 

o  CMA  and  MDA  DS  COI  vessel  track  data  were  84%  common, 
o  CMA  data  were  93%  common, 

o  MDA  DS  COI  were  81%  common  with  track  data  provided  via  the  legacy  COP. 
o  The  MDA  DS  COI  data  were  less  common  than  the  CMA  data  because  of 
interruption  in  input  data  streams  during  part  of  the  analyze  period, 
o  In  other  regions  (open  ocean)  MDA  DS  COI  picture  was  substantially  less 
complete  than  CMA  due  to  lack  of  classified  inputs, 
o  In  a  littoral  region,  CMA  data  latency  was  127  min  on  average  compared  with  24 
min  for  the  legacy  COP  and  40  min  for  MDA  DS  COI. 
o  CMA  track  data  latency  was  substantially  larger  in  an  open-ocean  area:  11.1  hrs 
on  average  for  the  analyzed  sample  of  tracks  in  the  Indian  Ocean, 
o  Because  of  the  CMA  data  latency,  the  current  positions  of  CMA  tacks  lag  the 
positions  held  in  the  legacy  COP.  The  capability  to  transfer  track  data  from  CMA 
to  the  legacy  DOP  was  demonstrated;  but  doing  so  resulted  in  a  redundant  track 
when  the  legacy  COP  already  held  the  contact.  The  legacy  system  does  not 
correlate/fuse  data  input  from  CMA. 

(7)  E-7  Communication  Capacity.  Can  existing  communications  paths  support  added  MDA 
Spiral-1  communications  requirements?  (GREEN) 

-  Although  not  a  specific  tasking,  the  communication  capacity  required  by  MDA  Spiral- 1  was 
assessed  only  to  make  a  determination  of  the  impact  on  future  installations  of  the  Spiral- 1 
capability.  A  degradation  in  capability  was  observed  at  on  node  due  to  bandwidth  limitations 
while  conducting  multiple  exercises.  The  node  was  required  to  timeshare  bandwidth  to 
accommodate  each  event.  Communications  capability  should  be  considered  if  there  are 
intentions  to  deploy  these  capabilities  afloat  or  to  a  shore-based  unit  with  limited  capabilities. 


C.  SUITABILITY  CRITICAL  OPERATIONAL  ISSUES 

(1)  S-1  Reliability.  (GREEN) 

•  The  Spiral- 1  suite  did  not  suffer  any  hardware  or  software  operational  mission  failures. 
The  MDA  suite  as  a  whole  was  operational  during  all  EARIGAME  exercise  periods. 

(2)  S-2  Maintainability.  (WHITE) 

•  The  NMIC  CMA  system  suffered  an  unknown  failure  that  rendered  it  unusable.  An 
attempt  by  NRL  to  troubleshoot  via  remote  access  was  unsuccessful.  A  technician 
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arrived  on-scene  to  conduct  on-site  troubleshooting  and  discovered  a  faulty  hardware 
router  and  replaced  it  five  hours  later. 

•  A  potential  issue  is  that  the  primary  means  of  troubleshooting  a  CMA  problem  is  for 
NRL  to  access  the  local  via  remote  web  access.  If  the  problem  cannot  be  resolved 
remotely,  NRL  must  dispatch  a  technician  to  the  node.  COMUSNAVCENT  however, 
has  a  permanent  SPAWAR  Fleet  Systems  Engineering  team  representative  onsite  to 
resolve  CMA  hardware  issues. 

•  The  Spiral- 1  suite  did  not  suffer  any  hardware  of  software  operational  mission  failures. 
The  MDA  suite  as  a  whole  was  operational  during  all  FAIRGAME  exercise  periods. 

(3)  S-3  Availability.  (GREEN) 

•  The  MDA  suite  was  operational  during  all  FAIRGAME  exercise  periods. 

(4)  S-4  Logistics  Supportability.  (GREEN) 

•  The  MDA  Spiral- 1  integrated  logistics  support  plan  (lESP)  was  reviewed.  No  issues 
were  observed. 

(5)  S-5  Compatibility.  (GREEN) 

•  No  compatibility  issues  were  observed. 

(6)  S-6  Interoperability.  (GREEN) 

•  Some  CMA  client  workstations  using  legacy  Internet  Explorer  6  could  not  operate  CMA. 
CMA  is  designed  to  operate  on  Internet  Explorer  7.  The  workaround  was  to  use  the 
Firefox  web  browser. 

•  CMA  requires  JAVA  1.6.  Other  host  systems  use  legacy  version  JAVA  1.4.  When 
CMA  is  loaded  on  the  same  hardware  as  GCCS-M,  the  JAVA  1.6  causes  the  local  GCCS 
workstation  to  crash.  The  workaround  is  that  CMA  must  be  loaded  on  separate  hardware. 

•  When  using  CME  and  Google  Earth  4.2  simultaneously,  system  crashes  resulted.  Google 
Earth  4.2  had  to  be  configured  to  use  DIRECTX  graphics  setting  instead  of  OpenGE 
graphics  setting  because  the  version  of  OpenGE  installed  on  the  CMA  workstation  causes 
Google  Earth  to  crash. 

(7)  S-7  Training.  (YEEEOW) 

•  The  post-installation  training  was  generic  in  nature  and  not  tailored  to  the  individual  site 
requirements/mission.  User  feedback  indicated  that  the  FAIRGAME  scenario  construct 
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provided  a  more  realistic  training  environment  because  the  scenarios  allowed  for  more 


real-world  applications  of  the  technologies.  An  official  Navy  training  pipeline  is  not  in 
place  roe  MDA  Spiral- 1. 

(8)  S-8  Human  Factors.  (GREEN) 

•  Because  MDA  Spiral- 1  was  not  a  composite  system,  users  were  required  to  switch 
between  windows  and  classification  domains  multiple  times,  causing  them,  at  times,  to 
become  confused  as  to  which  domain  they  were  looking  at.  The  confusion  caused  users 
to  lose  track  of  the  original  security  classification  of  the  information  they  wished  to 
disseminate. 

(9)  S-9  Safety.  (GREEN) 

•  No  safety  issues  were  observed. 

(10)  S- 10  Documentation.  (YEEEOW) 

•  Current  Standard  Operating  Procedures  (SOP)  and  TIP  do  not  address  the  new  MDA 
Spiral- 1  tools.  OPNAV  N3/N5  has  developed  an  outstanding  draft  MDA  SOP  that,  with 
further  refinement  and  extensive  Fleet  feedback  will  provide  the  groundwork  for  all 
MDA  nodes  to  employ  the  new  technologies  to  their  fullest  extent.  The  SOP  in  its 
current  version  will  assist  the  users  in  navigating  through  the  different  technologies,  but 
requires  feedback. 

•  Beyond  the  tasking  provided  by  the  White  Cell,  the  users  did  not  know  why  or  when  to 
use  a  specific  technology  to  produce  an  expected  results.  Users  required  prompting  by 
the  White  Cell,  and  observers,  to  think  outside  of  the  box  when  looking  for  specific 
information. 

•  Users  were  unaware  of  technology  manuals  and  the  Online  Learning  Development 
(OLLD).  The  draft  online  learning  compact  disc  was  made  available  to  the  nodes  no 
earlier  than  two  days  before  the  start  of  FAIRGAME.  The  draft  version  of  the  OLLD  is 
sufficient  to  provide  the  user  with  a  baseline  buttonology  level  of  knowledge.  The 
finalized  version  of  the  OLLD  is  expected  to  be  made  available  via  web  services 
sometime  in  August  2008. 

D.  OPERATIONAL  CONSIDERATIONS  FOR  LEAVE  BEHIND  TECHNOLOGIES 

(1)  CARGO  LINK. 
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•  During  Trident  Warrior  Cargo  Link  performed  very  well  during  seripted  seenarios 
facilitated  by  the  Subject  Matter  Expert  (SME).  Most  queries  found  the  expected  results 
on  the  first  attempt. 

•  During  EAIRGAME,  cargo  searches  at  one  node  did  not  produce  a  successful  query 
result  during  the  entire  event,  likely  due  to  insufficient  user  training. 

•  Cargo  Eink  contained  only  six  months  worth  of  data.  Searches  beyond  six  months 
require  a  request  to  the  NMIC  Global  Trader  office. 

•  Cargo  data  was  observed  in  CM  A  but  not  on  all  vessels. 

•  The  scope  of  partner  nations  with  cargo  sharing  agreements  is  not  known. 

•  Cargo  Eink  is  still  a  developmental  system  that  will  not  become  operational  until  20  Aug 
2008. 

(2)  CMA. 

•  During  EAIRGAME,  CMA  was  often  the  technology  of  choice  when  starting  a  search. 
Many  of  the  EAIRGAME  users  had  experience  in  previous  Operational  Demonstrations 
(OD)  as  well  as  in  Trident  Warrior  and  noted  that  system  information  reliability 
continued  to  be  a  concern. 

•  Users  doubted  the  completeness  of  the  CMA  database  and  used  other  systems  to  validate 
the  CMA  search  result.  It  is  unclear  to  the  user  what  data  is  actually  feeding  CMA;  users 
noted  the  need  for  comparable  data  as  available  with  the  legacy  systems. 

•  Pre-defined  alerts  were  not  representative  of  current  operational  needs.  CMA  is  not 
capable  of  providing  user-defined  alerts. 

•  Single  vessel  searches  sometimes  resulted  in  multiple  tracks,  each  with  different 
information  attached.  Analysis  concluded  that  some  of  the  search  results  were  inactive 
tracks. 

•  Advance  searches  were  not  intuitive. 

•  Each  node  maintains  its  own  CMA  database  with  no  two  nodes  having  the  same 
information.  Eor  example,  the  MIECPAC  CMA  database  contained  information  not 
available  to  the  PACEET  database. 

•  During  the  casualty  to  the  NMIC  CMA  server,  users  noted  excessive  search  lag  times 
when  conducting  queries. 
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•  At  times  CMA  would  exhibit  an  ERROR  500  message  whieh  required  the  user  to  exit  the 
web  browser  and  sign  back  in. 

•  Although  the  data  reliability  issue  cited  by  users  remains  a  concern,  CMA  is  still  a 
capable  system  and  greatly  enhances  the  warfighter’s  ability  to  maintain  situational 
awareness. 

(3)  FASTC2AP. 

•  User  feedback  indicated  FASTC2AP  on  SIPRNET  would  be  more  useful/capable  with 
additional  GENSER  track  history  feeds  not  available  on  CENTRIX. 

•  A  geospatial  option  would  allow  greater  situational  awareness  and  agent  building. 

•  FASTC2AP  provides  a  basic  user-defined  anomalous  detection  capability. 

(4)  GOOGLE  EARTH  FUSION  SERVICES. 

•  Google  Earth  fusion  services  provided  some  utility  but  was  viewed  as  redundant. 

•  Users  used  Google  Earth  to  display  vessels  of  interest  by  downloading  KML  files  from 
the  legacy  SEAWATCH  database. 

•  Google  Earth  fusion  services  4.2  was  not  observed  on  the  PACFLT  NMCI  SIPR  LAN, 
the  intended  network,  but  was  installed  and  observed  on  CENTRIX. 

(5)  LINX. 

•  User  feedback  indicates  limited  training  prior  to  Trident  Warrior. 

•  Other  than  the  collaboration  tools,  the  system  was  viewed  as  redundant  and  time 
consuming. 

•  The  full  potential  use  of  the  system  requires  a  shift  in  current  NCIS  investigative 
processes. 

•  Users  did  see  utility  in  the  Force  Protection  Portal  for  viewing  historical  incident  data  and 
for  its  collaborative  ability. 

(6)  MAGNET. 

•  Information  from  a  stand-alone  MAGNET  system  was  not  available  due  to  sharing 
agreements  concerning  U.S.  person’s  information. 

•  Vessel  information  provided  by  MAGNET  was  observed  via  CMA,  if  it  was  available. 

(7)  MDA  DS  COL 
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During  Trident  Warrior  the  demonstrations  were  facilitated  by  the  TW  MDA  Focus  Area 
lead. 


FAIRGAME  user  feedback  indicated  a  limited  utility  that  might  be  accessed  more  as  an 
afterthought  or  when  all  other  systems  had  been  exhausted. 


Anomalous  behavior  detection  capabilities  were  not  observed. 


(8) 


SEAPORT. 

SOVT  was  not  complete  prior  to  the  end  of  EAIRGAME  events. 


Test  data  upload  into  SEAPORT  was  not  observed,  however,  SPAWAR  SOVT  results 
indicate  a  successful  transfer  of  an  imagery  file  from  the  TES  to  SEAPORT. 


(9)  SMS/JPSC2. 


SMS  data  was  observed  in  the  MDA  DS  COT 


(10)  TAC. 

•  The  TAC  system  was  the  technology  of  choice  for  users  who  were  experienced  with  the 
capabilities  it  provides. 

•  Contextual  information  on  vessels  of  interest  and  persons  of  interest  is  outstanding. 

•  One  major  limitation  noted  was  the  range  of  historical  information  available.  Users 
indicated  that  data  currently  available  does  encompass  the  full  DIA  historical  database. 

•  Users  also  noted  that  certain  information  is  stripped  from  the  database  before  it  reaches 
the  user,  but  what  is  stripped  is  unknown. 

(11)  TES. 

•  The  TES  demonstration  was  conducted  per  the  developer  test  plan  in  conjunction  with  a 
U.S.  Navy  VBSS  graduation  scenario  pierside  onboard  the  USS  MOUNT  RUSHMORE 
(LSD  47).  Additional  developer  end-to-end  testing  was  conducted  at 
COMUSNAVCENT  ICW  SOVT. 


10.7  Findings 

MDA  Spiral- 1  is  determined  to  be  potentially  operationally  effective  and  potentially 
operationally  suitable. 

The  leave-behind  technologies  will  enhance  the  warfighter’s  ability  in  maintaining  situational 
awareness  of  the  global  maritime  domain. 
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Current  MDA  training  and  documentation  provide  the  basic  building  blocks  to  employ  the 
Spiral- 1  suite. 


Additionally,  the  warfighter’s  ability  to  locate,  track,  an  pass  information  to  other  MDA  nodes  is 
enhanced. 


10.8  Recommendations 

A.  OPNAV  consider  implementing  Predictive  Analysis  for  Naval  Deployment  Activity 
(PANDA)  and  Maritime  Auto  Super  Track  Enhance  Reporting  (MASTER)  as  MDA 
capabilities.  Although  not  Spiral- 1  technologies,  the  OTA  did  observe  both  demonstrations 
during  Trident  Warrior,  and  while  still  in  the  developmental  stage,  both  technologies 
demonstrated  utility  in  filling  capability  gaps  in  regards  to  vessel  tracking  and  capturing/storing 
maritime  baseline  movement  patterns  in  order  to  provide  anomalous  detection. 

B.  OPNAV  resolve  information  sharing  agreements  concerning  U.S.  persons. 

C.  OPNAV  convene  an  MDA  users  working  group,  to  include  coalition  participation,  to  refine 
MDA  documentation  requirements. 

D.  OPNAV  implement  a  dedicated  MDA  course  of  instruction  of  include  it  in  existing  NTSPS. 

E.  NCIS  continue  to  refine  CNIC  MDA  CONOPS  and  conduct  additional  law  enforcement 
exercises  using  real-world  data  to  educate  agents  on  the  full  capabilities  of  EINX. 

E.  PEO  C4I  consider  deploying  EASTC2AP  on  SIPRNET  and  expanding  the  capability  to  other 
nodes. 

G.  PEO  C4I  track  estimated  Cargo  Eink  operational  date  and  node  user  account  set-up. 

H.  PEO  C4I  conduct  additional  advanced  training  in  an  operational  setting  using  real-world  data 
is  required  to  maximize  capability  utility. 

I.  PEO  C4I  investigate  and  resolve  the  cause  of  Google  Earth  4.2  installation  issues  at 
COMPACEET. 

J.  PEO  C4I  finalize  OEED  and  make  it  available  via  web  access. 

K.  PEO  C4I  improve  human-system  integration  in  conjunction  with  development  of  follow-on 
spirals. 

E.  PEO  C4I  allows  adequate  time  between  installation  and  future  assessment  events  to  allow 
users  to  become  proficient  at  using  the  new  tools. 
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M.  NRL  investigate  the  CMA  IE7/Firefox  web  browser  and  JAVA  1.4/1. 6  issue. 


10.9  Caveat 

This  QRA  presents  primarily  statements  of  opinion  and  not  findings  of  fact.  Operational  Test 
and  Evaluation  is  required  to  substantiate  the  results  of  this  assessment. 

10.10  FAIRGAME  Survey  Results 

Special  surveys  for  EAIRGAME  operators  have  been  are  being  executed.  The  results  will  be 
available  at  a  later  time  and  presented  in  a  subsequent  version  of  this  report. 

D,3  White-Cell  Logs  and  Chat  Results 

White-Cell  chat  logs  are  classified  and  cannot  be  included  in  this  report.  Non-classified  results 
have  been  extracted  and  are  presented  in  Sections  5  through  8. 


113 


11.0  Appendix  E:  Technical  Risk  Reduction  Limited  Objective 
Experiment  Results 

TRRLOE  was  a  risk  reduction  limited  objective  experiment  for  FAIRGAME.  It  executed  the 
FAIRGAME  defined  events,  although  without  the  full  suite  of  Spiral- 1  systems.  It  was 
conducted  in  a  laboratory  environment  rather  than  at  operating  centers.  The  experiment  was 
carried  out  in  Lab  140,  SSC,  San  Diego. 

Three  users  participated  in  the  TRRLOE;  two  participants  used  the  CMA  technology  and  one 
used  FASTC2AP  and  Google  Earth  systems  throughout  the  experiment.  Participants  included  an 
IS-3  Strike  Analyst,  an  IS-2  Optical  Analyst,  and  a  LT  Intelligence  Analyst,  currently  attending 
the  Naval  Postgraduate  School.  The  average  service  time  for  the  participants  was  about  five 
years  and  their  level  of  experience  in  their  current  positions  ranged  between  intermediate  and 
advanced. 

These  results  are  based  on  extractions  of  data  from: 

•  Observer  logs 

•  Hot-wash-ups  that  were  conducted  with  all  personnel  at  the  end  of  each  day 

•  Surveys  that  were  completed  by  the  operators 

The  following  results  are  summaries  obtained  directly  from  the  data  sources,  prior  to  detailed 
analysis. 

11.1  Observer  Log  Results 

The  following  results  are  for  CMA,  except  where  noted. 

Information  Acquisition  Times  -  Even  though  the  systems  were  not  used  in  an  operational 
environment,  the  amount  of  time  required  to  obtain  information  using  them  was  representative. 
Very  little  time  was  required.  E.g., 

•  1  min  to  determine  when  ship  was  built  after  White  Cell  request. 

•  7  min  to  determine  person-of-interest  was  not  on  reported  ship. 

•  2  min  to  identify  cargo  on  a  named  ship. 

•  10  min  to  locate  a  ship  and  identify  its  origin  and  arrival  locations. 

•  6  min  to  locate  a  ship,  details  on  type  and  crew. 

It  is  possible  that  these  times  are  biased  to  shorter  time  because  historical  data  was  used  for 
which  the  needed  data  was  known  to  be  present.  Even  so,  these  information  acquisition  times 
represent  an  improvement  over  existing  procedures. 


114 


Operator  Performance/! raining  -  Operator  training  on  the  operations  they  were  asked  to 
perform  with  the  systems  was  minimal.  Within  this  eontext,  the  following  observations  ean  be 
made: 

•  They  developed  faeility  with  the  systems  in  a  short  period  of  time,  usually  after  about  14 
hour  of  working  with  a  funetionality. 

•  How  the  system  applied  to,  or  eould  be  used  for,  the  jobs  with  whieh  they  were  familiar 
was  elear.  Diffieulties  they  had  were  largely  the  result  of  system  malfunetion  (due  to  lab 
environment). 

•  There  were  instanees  where  an  operator  searehed  for  a  funetion  or  eapability  CMA  did 
not  have.  This  eould  be  a  training  issue  and/or  represent  functionalities  that  should  be 
added. 

•  It  is  clear  that  two  or  three  days  of  focused  training  on  the  Spiral- 1  suite  will  result  in 
fully  competent  operators. 

Human-System  Interaction  -  Note  that  the  following  HSI  observations  are  based  on  operators 
who  were  not  fully  trained  on  the  systems.  More  complete,  operationally-based,  training  would 
alleviate  some  of  the  difficulties  encountered.  These  observations  apply  to  CMA  unless 
otherwise  noted. 

•  The  layout  of  advanced  search  is  cumbersome  (layout  and  search  parameters  disappear 
when  new  search  parameters  are  entered). 

•  The  search  information  used  when  initiating  a  search  cannot  be  seen  when  scrolling  down 
in  the  display.  This  can  result  in  losing  track  of  overall  search  context.  Operators 
requested  a  horizontal  rather  than  vertical  layout  of  information. 

•  Having  political  and  AOR  boundaries  would  aid  the  operator  in  setting  up  search  areas. 

•  Searching  for  and  acquiring  information  can  involve  many  information  elements.  It 
would  be  useful  to  have  a  pre-structured  notepad  where  the  operator  could  enter 
information  already  obtained  and  build  an  information  picture. 

•  Operators  tend  to  develop  personalized  methods  for  information  searches.  It  would  be 
useful  to  have  a  means  for  capturing  operator  searches  for  reuse.  This  would  be 
especially  useful  for  categories  in  advanced  search. 

•  Add  ‘cargo’  to  pull  down  search  menu. 

•  FASTC2AP:  The  structure  of  the  format  used  for  setting  up  alerts  is  unclear  and  difficult 
to  use. 

•  FASTC2AP:  Add  capability  to  make  a  polygon,  a  common  shape  required  to  conduct  a 
search  in  the  real  world. 

•  Add  feature  to  specify  a  point  and  a  specified  radius  to  establish  a  search  area. 

•  Add  feature  to  be  able  to  enter  a  coordinate  and  then  be  able  to  set  up  a  search  area 
around  it. 

•  Need  better  color  coding  along  left-hand  side  for  icons;  make  meaning  of  color  explicit. 

•  Need  better  understanding  of  what  pre-built  searches  do;  user  thought  they  were  going  to 
provide  different  information. 

•  Need  ability  to  save  an  area  when  drawing  a  box/polygon  with  a  specific  name. 

•  Change  hyper-graph:  provide  more  detailed  information  (e.g.,  vessel  name)  and  make  it 
easier  to  read  information 
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•  Include  the  ability  to  draw  a  line  to  be  able  to  seareh  any  track  crossing  a  line  rather  than 
a  box. 

System  Performance  -  Most  system  problems  encountered  were  due  to  the  lab  environment  and 
do  not  qualify  as  Spiral- 1  system  results.  The  following  results  are  inherent  to  the  systems,  not 
the  lab  environment.  These  results  are  derived  from  limited  tests  of  the  systems  and  have  not 
been  validated. 

•  Partial  mateh  did  not  work  as  antieipated  when  searehing  for  a  ship  name. 

•  Operators  eould  not  zoom  in  and  draw  a  small  geographieal  box  around  a  VOI. 

•  Advanced  search  is  difficult  to  use.  Speeifications  can  be  lost  when  adding  to  seareh. 

•  Basic  and  advanced  seareh  need  more  defined  categories,  sueh  as  “hazmat”  for  cargo. 

•  It  would  be  advantageous  to  be  able  to  use  distances  when  setting  up  searehes. 

•  FASTC2AP;  Three  of  the  alerts  (Course  Proximity,  Vessel  Proximity,  and  Next 
Waypoint  Feasible,  require  the  SCONUM.  Matching  of  ship  name,  position  and 
SCONUM  becomes  elassified.  Since  FASTC2AP  will  be  fielded  on  CENTRIX,  perhaps 
the  ship’s  Lloyd’s  or  International  Maritime  Organization  (IMO)  number  could  be 
substituted  for  the  currently  required  SCONUM. 

•  FASTC2AP:  Boolean  criteria  alerts  were  difficult  to  set  up  and  all  eonditions  may  not  be 
available. 

•  FASTCAP;  Needs  eapability  to  refine  searches  more  by  user:  current  searches  are  too 
broad. 

•  FASTCAP:  Understanding  the  results  of  agents  was  diffieult 

•  FASTCAP:  Could  not  designate  an  agent  to  find  ships  that  came  within  200nm  proximity 
of  the  west  coast. 

•  FASTCAP:  Flexible  template  relations  were  difficult  to  understand. 

•  FASTCAP:  Interface  was  not  intuitive  to  use. 

•  Need  separate  category  to  rate  the  age  of  the  data  -  an  important  criteria  on  which 
operational  tasking  is  built  on.  (There  is  a  quality  rating,  but  time  lateness  of  data  should 
be  a  separate  category.) 

•  Make  the  factors  that  go  into  “quality”  rating  transparent  to  the  user. 

•  Need  a  clear  indieator  to  inform  user  when  any  database  is  down. 

•  GOOGLE  EARTH:  Latest  position  was  easy  to  get,  but  the  Volpe  AIS  data  was  limited 
and  not  useful  for  in-depth  analysis. 

•  GOOGLE  EARTH:  Could  search  for  the  latest  position,  but  not  for  track  data. 


11.2  Survey  Results 

Surveys  were  administered  for  each  of  the  technologies  involved  in  the  TRRLOE  in  addition  to 
demographics  and  training  surveys.  System  specific  survey  questions  were  developed  in  addition 
to  the  general  survey  questions  that  appear  in  all  three  surveys.  Surveys  used  for  TRRLOE  MDA 
technologies  are  included  in  Appendix  A.  The  purpose  of  the  surveys  was  to  gather  feedback 
from  the  users  involved  in  the  exereise  to  identify  strengths  and  weaknesses  of  the  MDA 
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technologies.  Topics  included  training  effectiveness,  system  usability,  suggestions  for 
improvement,  and  features  participants  found  particularly  useful. 

Baseline  Training  Survey  Results 

All  three  system  operators  had  no  prior  knowledge  of  Comprehensive  Maritime  Awareness 
(CMA)  or  FASTC2AP  technologies,  or  their  capabilities,  prior  to  the  training.  All  users  received 
an  instructional  brief,  and  depending  on  the  participant,  some  received  hands-on  training,  a  quick 
reference  guide,  and  trainer  interaction.  Two  users  felt  that  the  training  materials  were  easy  to 
understand  given  the  amount  of  time  allowed  to  complete  the  tasking,  and  the  other  felt  that  the 
material  was  not  easy  to  sift  through.  Participants  thought  the  preparation,  instruction,  and 
training  provided  during  TRRLOE  for  CMA  and  FASTC2AP  was  fairly  productive,  all  users  felt 
that  there  should  have  been  more  time  allotted  in  order  to  adequately  understand  the  processes 
and  systems  as  a  whole,  in  addition  to  working  one-on-one  with  subject  matter  experts. 

Regarding  Google  Earth,  all  users  had  prior  knowledge  of  its  capabilities.  All  users  thought  the 
instructional  briefs  and  on-line  tutorials  were  easy  to  understand  and  also  agreed  the  training  and 
instruction  received  prepared  them  for  the  tasks  performed  during  TRRLOE. 

Comprehensive  Maritime  Awareness  (CMA) 

Regarding  the  training  materials  provided,  one  user  felt  that  for  the  limited  time  allotted,  the 
materials  made  it  fairly  easy  to  complete  the  tasking  while  the  other  user  felt  that  the  training 
materials  were  not  easy  enough  to  navigate  through  and  suggested  an  online  version  to  improve 
the  method  of  searching.  One  participant  felt  strongly  about  allowing  more  time  during  the 
exercise  in  order  to  adequately  comprehend  the  material,  while  the  other  user’s  concern  was  to 
provide  more  one-on-one  time  with  operators  and  subject-matter  experts.  One  participant 
suggested  that  the  system  should  have  been  fully  developed  beforehand  to  enable  the  participants 
to  know  what  to  focus  on  during  the  training  and  get  the  most  out  of  the  exercise  along  with 
more  scenarios  in  order  to  practice  on  prior  to  the  test  evaluations.  Although  complications  were 
experienced  during  the  exercise,  overall,  the  users  felt  satisfied  with  the  training  they  received. 

FASTC2AP 

The  FASTC2AP  operator  had  approximately  one  hour  of  training  on  the  system  prior  to  the 
exercise,  where  he  was  first  introduced  to  the  technology.  Two  operators  received  training  on  the 
system  in  the  form  of  an  instructional  brief  which  they  both  agreed  was  easy  to  understand. 
Participants  agreed  the  training  was  effective,  but  it  could  have  been  longer,  and  suggested  that 
more  scenarios  be  added  to  the  exercise  in  order  to  practice  more  before  evaluations,  and  also 
requested  more  one-on-one  time  with  operators. 

Change  Alerts  so  SCONUM  is  Not  Required.  FASTC2AP  has  a  number  of  pre-defined  alert 
functions.  Their  use  requires  an  ONI-assigned  SCONUM,  which  when  combined  with  a  ship’s 
name  and  its  position  becomes  classified.  The  SCONUM  also  may  not  be  available  to  coalition 
forces.  This  makes  these  alerts  of  limited  use  and  could  cause  future  security  concerns.  It  is 
recommended  that  the  alerts  be  rewritten  to  require  ship  names  only  or  the  Lloyd’s  identification 
number. 
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GOOGLE  EARTH 

For  the  Google  Earth  application,  all  system  operators  had  prior  knowledge  of  the  technology. 
Participants  received  an  on-line  tutorial  in  addition  to  an  instructional  briefing  that  they  felt  was 
easy  to  understand  and  use;  the  training  was  adequate  for  them  to  effectively  utilize  the 
technology.  Due  to  the  familiarity  of  the  application,  the  users  were  satisfied  with  the  training 
they  received  and  felt  that  they  could  use  Google  Earth’s  technology  effectively  by  the  end  of  the 
exercise. 


Comprehensive  Maritime  Awareness  (CMA)  Survey 

Usability.  Both  CMA  operators  thought  the  technology  was  easy  to  learn  and  use,  and  the 
training  produced  basic  knowledge  of  the  system  and  its  background.  Both  CMA  users  strongly 
agreed  it  was  easy  to  develop  and  maintain  situation  awareness  on  a  vessel  of  interest  (VOI), 
especially  once  provided  with  a  watch  list  that  includes  the  names  of  the  vessels.  CMA  operators 
emphasized  that  it  was  extremely  easy  to  find  relevant  information  on  a  VOI  as  long  as  one  knew 
the  name  of  the  vessel;  otherwise  it  would  be  difficult  due  to  the  hyper  graph. 

It  was  easy  for  CMA  users  to  associate  tipper/  Intel  information  with  the  correct  VOI.  CMA 
showed  recent  alerts  on  the  vessel  and  made  it  easy  to  see  the  tippers.  Operators  maintained 
contact  with  the  white  cell  (representing  higher  command)  on  chat.  On  day  one,  users  found  it 
difficult  to  cross  reference  tracks  being  used  by  operators  on  other  systems  due  to  the  difficulty 
of  looking  at  two  units  and  not  being  able  to  compare  them.  However,  on  day  two,  CMA  users 
agreed  that  the  use  of  either  EASTC2AP  or  Google  Earth  made  it  much  easier  to  cross  reference 
and  pull  up  the  necessary  information  to  collaborate. 

Suggestions  for  Improvement  and  Useful  Features.  Both  CMA  users  experienced  difficulties 
using  the  alerting  system  as  well  as  the  hyper  graph.  Participants  noted  that  when  searching  on 
CMA  using  a  partial  search,  the  system  did  not  always  pull  up  all  tracks  containing  a  portion  of 
the  name.  One  user  recommended  adding  a  feature  to  provide  the  capability  to  use  range  and 
bearing  from  a  point  in  a  circle  to  form  a  search  area.  This  user  also  suggested  including  the 
ability  to  use  a  line  or  border  to  search  for  any  track  crossing  a  line  rather  than  a  box  as  is  done  in 
the  current  system. 

Another  user  noted  that  when  attempting  to  save  a  box  or  area,  the  program  would  not  warn  the 
user  when  an  existing  file  had  the  same  file  name,  causing  confusion  when  referring  back  to  a 
basic  search  in  the  pull  down  menu,  and  finding  two  files  with  the  same  name.  The  user  also 
suggested  that  the  system  should  include  a  search  pull  down  in  the  metadata  section  that  allows 
for  the  search  of  cargo,  making  it  easier  to  narrow  down  the  hits.  CMA  users  stated  that  the 
scenarios  were  realistic  enough,  and  foresee  the  CMA  system  becoming  a  very  useful  tool  for  all 
AORs  once  incorporated.  Participants  thought  conducting  analysis  on  a  VOI  was  easier  and 
much  faster  when  using  CMA  than  with  their  previous  system. 

FASTC2AP 
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Usability.  The  participant  felt  that  the  program  was  difficult  to  learn  resulting  in  the  user  having 
to  try  the  agents  multiple  times  in  order  to  understand  what  each  was  providing  in  the  way  of 
results.  On  the  second  day  the  FASTC2AP  operator  stated  it  was  fairly  easy  to  create  agents, 
however  understanding  the  results  was  still  difficult.  This  user  was  not  able  to  utilize  the  online 
aid,  nor  did  he  know  what  the  software  “wizards”  were.  The  FASTC2AP  operator  thought  it  was 
easy  to  compose  agents,  although  the  system  could  not  provide  the  details  required  for  some  of 
the  RFIs.  This  user  found  it  very  useful  to  build  alerts  in  order  to  fill  RFIs  through  the 
FASTC2AP  alerting  system.  Since  this  exercise  only  looked  at  snapshot  type  problems,  it  was 
hard  to  maintain  situation  awareness  on  a  VOI  given  that  they  did  not  track  a  VOI  over  a  period 
of  time.  When  collaborating  with  other  operators  during  the  scenario,  the  operator  was  able  to 
cross  reference  some  tracks,  however  CMA  and  FASTC2AP  had  access  to  different  data,  making 
it  somewhat  difficult. 

Suggestions  for  Improvement  and  Useful  Features.  At  one  point  this  user  wanted  to  find  ships 
that  came  within  200  NM  proximity  of  the  west  coast,  and  he  was  not  able  to  designate  an  agent 
to  carry  out  the  task,  suggesting  that  this  capability  should  be  added.  This  user  also  added  that 
the  flexible  template’s  relations  were  difficult  to  understand  and  suggested  that  the  graphical  user 
interface  (GUI)  be  made  more  intuitive.  Lastly,  this  participant  suggested  removing  the  ship 
control  number  as  a  required  field  when  using  it  on  an  unclassified  network.  Overall,  the  user 
viewed  FASTC2AP  as  an  effective  VOI  analysis  tool  with  the  correct  data  to  support  it. 


GOOGLE  EARTH 

Usability.  The  user  felt  that  this  application  was  easy  to  learn,  and  use,  finding  no  difficulty  in 
using  all  of  the  features  included  in  Google  Earth.  This  user  stated  the  system  was  a  quick 
geospatial  reference  for  developing  awareness  on  a  VOI  as  well  as  finding  information  on  them. 
Conducting  analysis  on  a  VOI  was  easy,  specifically  for  acquiring  their  last  position;  however 
the  VOLPE  AIS  data  was  limited,  and  not  useful  for  in-depth  analysis  or  for  providing  tracking 
data.  There  was  no  embedded  collaboration  tool  included,  thus  it  was  not  possible  to  send 
information  from  Google  Earth  to  other  systems.  However  it  was  possible  to  look  up  vessels  in 
other  systems  making  it  easier  to  cross  reference  tracks  and  information  on  VOI  with  other 
operators. 

Suggestions  for  Improvement  and  Useful  Features.  The  user  found  it  useful  as  a  platform  for 
displaying  data  and  not  as  a  collaboration  tool.  Other  than  the  missing  information  due  to  the 
difference  in  databases  and  the  restrictions  of  the  application,  the  user  found  the  system  to  be 
very  helpful  throughout  the  exercise.  The  feature  this  participant  found  most  helpful  was  the 
“find”  option,  providing  an  easy  way  to  see  if  a  VOI  was  in  the  data  feed  as  well  as  finding  basic 
information  on  the  VOI. 
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12.0  Appendix  F  :  Empire  Challenge  2008  (EC08)  Results 


Empire  Challenge  is  a  yearly  ISR  experiment  sponsored  by  NGA  and  JFCOM.  Although  it  did 
not  perform  dedieated,  MDA  events,  there  were  three  MDA  related  events: 

o  Land-based  biometrie  eolleetion,  and 

o  Sea-based  biometrie  eolleetion. 

o  Vessel  traeking  with  a  P3, 

Information  extraeted  from  the  first  two  events  is  reported  here,  not  a  eomplete  description  of  the 
events  to  avoid  superfluous  discussion.. 

12.1  Land-Based  Biometric  Collection 

China  Lake  biometric  collections  events  were  conducted  on  July  23  and  29.  For  each  date,  latent 
prints  were  collected  from  an  Opposing  Force  (OPFOR)  vehicle  and  a  full  spectrum  of  biometric 
collections  was  performed  on  a  subject  located  in  a  cave. 

Latent  Prints 

Initial  collections  were  made  after  an  OPFOR  vehicle  was  engaged  and  disabled.  The  steps  in 
the  collection  and  analysis  process  were: 

•  Two  prints  collected  from  the  vehicle  were  digitized  with  a  ScanShell  1000  scanner. 

•  The  Remote  Latent  Submission  laptop  embedded  the  digitized  prints  in  Electronic 
Biometric  Transmission  Specification  (EBTS)  messages  using  the  Videntity  Latent 
Alchemist  application. 

•  The  biometric  files  were  transferred  to  a  laptop  on  the  Battlespace  Awareness  ISR 
Integration  Capability  (JBAIIC)  vehicle  via  flash  drive. 

•  The  files  were  transmitted  to  the  Biometric  Fusion  Center  (BFC)  via  the  China  Lake 
Community  Of  Interest  (COI)  network,  the  Joint  Mission  Support  Module  (JMSM),  and 
the  Tactical  Network  Topology  (TNT)  network. 

•  BFC  conducted  the  match  assessment  and  placed  the  report  on  the  Video  Conference  tool 
#1(VC1)  file  repository. 

Cave  Collection 

The  biometric  collection  inside  the  cave  was  carried  out  as  follows: 

•  Collection  was  with  the  Crossmatch  SeekID  collection  device. 

•  The  enrollment  consisted  of  10  finger  prints,  two  iris  images  and  one  facial  image. 

•  The  biometric  files  were  transmitted  from  the  SeekID  device  in  the  interior  of  the  cave  to 
the  JBAIIC  vehicle  parked  just  outside  the  cave  by  a  sequence  of  three  Cheetah  radios. 

•  The  messages  were  then  passed  over  the  COI  network  to  the  JMSM  and  from  there  over 
the  TNT  network  to  the  BFC. 

•  The  BFC  developed  a  report  (match/no-match)  on  the  biometric  collection  which  they 
published  to  a  File  Transfer  Protocol  (FTP)  site. 
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•  The  Office  of  Naval  Intelligence  (ONI)  did  not  participate  in  the  cave  biometric 

collection  scenario,  therefore  the  BFC  provided  the  tasking  (detain/  release)  response  to 
the  biometric  team. 


Biometrics  in  the  Common  Tactical  Picture  (CTP) 

The  BFC  generated  Cursor  on  target  (CoT)  messages  for  the  collection/enrollment,  identification 
(match/no-match),  and  tasking  (detain/  release)  biometric  events.  The  procedure  to  provide  CTP 
information  was: 

•  The  CoT  messages  were  sent  from  the  BFC  via  TNT  to  the  JMSM  TNT  node  where  they 
were  pushed  through  a  firewall  to  the  COI  unclassified  network. 

•  The  display  of  these  events  in  the  CTP  requires  that  the  location  of  the  event  be  included 
in  the  message. 

•  There  were  no  positions  provided  with  the  latent  print  messages  and  for  the  cave 
collection,  there  was  no  SeekID  Global  Positioning  System  (GPS)  reception  and  therefore 
no  position  was  included  in  the  biometric  messages. 

•  Accordingly,  the  BFC  manually  entered  into  the  CoT  messages  the  position  reported  to 
them  over  VCl  chat. 

•  The  CoT  messages  were  disseminated  by  the  COI  CoT  server  and  successfully  displayed 
in  the  COI  CTP,  passed  through  the  Raytheon  High  Speed  Guard  (RHSG),  and  displayed 
in  the  CFE  CTP. 
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Figure  7.  COI  CTP  Screen  Capture  of  Biometrics  Events 


This  Google  Earth  screen  capture  shows  Ripper,  the  JBAIIC  vehicle,  whose  position  was 
provided  by  the  PRC-1 17G  radio,  in  the  vicinity  of  the  cave.  The  red,  cross  icon  represents  the 
biometric  CoT  tasking  event.  It  is  superposed  on  a  red  triangle  icon  that  denotes  the  biometric 
CoT  identification  event,  EalconOb  is  the  JMSM  location. 


Findings 

•  The  China  Lake  biometric  events  were  fundamentally  successful.  The  prime  objective 
of  the  EC08  EMIO  event  was  to  demonstrate  transfer  of  appropriate  data  between  the 
various  participating  nodes.  Speed  of  execution  was  not  a  primary  concern. 

•  During  the  cave  biometric  collections  communication  to  the  BEC  were  lost  or  slow 
primarily  as  a  result  of  communication  unreliability  between  the  JBAIIC  vehicle  and  the 
Slate  Ridge  antenna.  During  the  July  29  latent  print  collection,  a  router  failure  delayed 
transmission  of  the  print  files  to  the  BEC. 
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•  The  biometric  coordinator  in  the  JMSM  at  China  Lake  had  adequate  Situation  Awareness 
(SA)  of  the  biometric  collection  and  assessment  process  through  VCl  chat.  Video  added 
little  in  part  because  of  the  problematic  communication  link. 

•  A  protocol  needed  to  be  defined  for  where  the  biometric  collections  are  to  be  posted  and 
to  where  BFC  results  are  to  be  disseminated. 

Assessment  of  EC08  China  Lake  Biometric  Events  Objectives 

Listed  below  are  the  JBAIIC  EC08  objective  questions  that  relate  to  the  China  Lake  biometric 
events  and  an  assessment  of  the  degree  to  which  the  objectives  were  satisfied. 

Q-1 :  Can  biometric  collection  messages  be  transmitted  from  the  interior  of  a  cave  to  the 
BFC  and  JMSM?  (Partially  Satisfied) 

Biometric  collection  messages  were  successfully  transmitted  via  Cheetah  radios,  the  COI 
network,  and  the  TNT  network  from  the  interior  of  the  cave  in  the  vicinity  of  CP  82  located  west 
of  China  Lake  Echo  range  to  the  BFC.  These  communications  were  not  consistent  because  of  a 
marginal  radio  link  between  the  vicinity  of  the  cave  and  Slate  Ridge 

Unlike  the  EMIO  demonstration,  the  biometric  files  were  not  posted  to  the  VCl  File  Repository 
so  that  the  collections  were  not  accessible  to  the  biometric  coordinator  in  the  JMSM. 

Q-2:  Can  biometric  match/no  match  messages  be  transmitted  from  the  BFC  to  the  cave 
biometric  collection  party  in  the  interior  of  the  cave  and  the  JMSM?  (Fully  Satisfied) 

The  match/no-match  messages  from  the  BFC  were  posted  to  the  VCl  File  repository  and/or  VCl 
chat  where  they  were  accessible  to  the  collection  party  and  the  biometric  coordinator  at  the 
JMSM. 


Q-3:  Can  biometric  detain  messages  be  transmitted  from  ONI/NMIC  to  the  cave 
biometric  collection  party  in  the  interior  of  the  cave  and  the  JMSM?  (No  Test) 

ONI  did  not  participate  in  the  China  Fake  biometric  events.  BFC  developed  the  detain  messages 
which  were  communicated  only  as  CoT  messages  to  the  JBAIIC  CTP. 

Q-4:  Were  the  biometric  collection,  match/no-match,  and  detain  events  displayed  in  the 
CTP?  (Fully  Satisfied) 

Biometric  CoT  messages  were  developed  by  the  BFC  and  sent  via  TNT  to  the  CFMCC  TNT 
node  in  the  JMSM  where  they  were  automatically  pushed  through  a  firewall  to  the  COI  network. 
The  CoT  messages  were  disseminated  by  the  COI  CoT  server  and  successfully  displayed  in  the 
COI  CTP,  passed  through  the  RHSG,  and  displayed  in  the  CFE  CTP.  For  the  July  29  cave 
biometric  event  all  three  biometric  CoT  messages  were  displayed  in  the  CTP. 
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Q-5;  Did  the  tools  provided  to  the  CFMCC  in  the  JMSM  provide  adequate  SA  to 
exereise  effeetive  C2  of  biometrie  events?  (Fully  Satisfied) 

The  CFMCC/biometrie  eoordinator  in  the  JMSM  at  China  Lake  had  adequate  SA  of  the 
biometrie  eolleetion  and  assessment  proeess  primarily  through  VCl  ehat.  Video  added  little 
beeause  of  the  very  low  update  rate. 


12.2  Sea-Based  Biometrie  Colleetion 

The  EMIO  biometrie  colleetion  was  conducted  aboard  the  Liberty  ship  Jeremiah  O’Brien  located 
in  San  Francisco  Bay  on  July  18,  2008. 

EMIO  Biometric  Data  Capture 

The  Biometric  collections  (enrollments)  were  performed  on  three  subjects  in  the  interior  of  the 
Liberty  ship  (aft  steering  room)  using 

•  Crossmatch  Jump  Kit  collection  devices  for  enrollment.  The  Jump  Kit  included: 

o  Panasonic  Toughbook, 
o  Guardian  R  Fingerprint  Capture  Device, 
o  I  Scan  Iris  Capture  Device, 
o  Cannon  A640  Digital  Camera,  and 

o  Mission  Oriented  Biometric  Software  (MOBS)  VI. 3. 3  software. 

The  enrollment  included: 

•  demographics, 

•  iris  image  capture  (both  eyes), 

•  10  fingerprints,  and 

•  full  facial  image. 

Biometric  Data  Sharing 

Data  and  information  sharing  were  as  follows: 

•  Biometric  collections  were  passed  as  Electronic  Eile  Transmission  (EET)  files  to  a  Eile 
Transfer  Protocol  (ETP)  site  at  the  Biometric  Eusion  Center  (BEC). 

•  Data  were  processed  with  the  Validation  And  Match  Tool  (VAMT). 

•  Match  results  were  passed  from  the  BEC  to  the  boarding  party  via  ETP,  to  the  CEMCC 
via  chat,  and  to  the  Office  of  Naval  Intelligence  (ONI)  via  e-mail. 

•  The  BEC  fusion  center  generated  Cursor  on  Target  (CoT)  messages  for  the  biometric 
events  for  display  in  the  JBAIIC  Common  Tactical  Picture  (CTP). 

•  ONI  generated  a  quick  look  report.  Biometric  Intelligence  Analysis  Reports  (BIAR)  and 
modified  BIARs,  which  were  passed  to  the  CEMCC  via  e-mail. 

•  The  CEMCC  forwarded  the  ONI  quick  look  report  to  the  Boarding  Officer  (BO)  via  chat. 


Communications 
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•  Cheetah  radios  were  used  for  the  transmission  of  data  between  the  interior  and  the 
Network  Operations  Center  (NOC)  of  the  ship  on  which  the  collection  took  place. 

•  A  sequence  of  six  Cheetah  radios  was  used  to  transfer  communications  between  the 
interior  and  the  NOC  (wardroom)  of  the  ship. 

•  From  the  NOC,  the  communications  flowed  through  a  hardwire  link  to  a  Tactical 
Network  Topology  (TNT)  node  radio  above  deck  on  the  ship.  The  TNT  network 
included  nodes  at: 

o  Jeremiah  O’Brian 
o  NPSTOC/NOC 
o  CFMCC 
o  BFC 

ONI  was  not  on  the  TNT  network  and  communications  between  ONI  the  BFC  and  the  CFMCC 
were  primarily  by  internet  e-mail. 

Collaboration:  CVl,  a  collaborative  application  developed  at  NFS,  was  the  principal  tool  used 
for  collaboration  in  the  EMIO  collection  demonstration.  CVl  rode  on  the  TNT  Virtual  Private 
Network  (VPN).  The  nodes  listed  in  the  following  table  participated  in  VCl  chat. 


Node 

Location 

CFMCC 

JMSM  2,  China  Lake 

NFS  NOC 

NFS,  Monterey 

SS  Liberty 

Jeremiah  O’Brien,  TOC/NOC.  Ward  room 

Boarding  Officer 

Jeremiah  O’Brien,  armory,  site  of  biometric  collection 

biocollect 

Jeremiah  O’Brien,  armory,  site  of  biometric  collection 

BFC 

Clarksburg,  West  Virginia 

Video:  Video  was  available  over  VCl .  The  only  video  source  was  from  the  engine  room  of  the 
Liberty  ship.  The  video  reached  VCl  via  cell  phone  not  the  Cheetah  radios.  The  video  was 
sporadic  and  of  limited  usefulness. 

File  Repository:  The  File  Repository  feature  of  VCl  allows  users  to  place  files  in  a  repository 
where  they  can  be  retrieved  by  other  user. 


Biometrics  in  the  Common  Tactical  Picture  (CTP) 

The  BFC  generated  CoT  messages  for  and  the  match/no-match  events  (CoT  event  type  b-a).  The 
intent  was  also  to  develop  CoT  messages  for  the  collection/enrollment  and  the  ONI  tasking 
(detain,  not  detain)  events  but  this  was  not  realized  in  the  EMIO  event. 

•  The  CoT  messages  were  sent  via  TNT  to  the  CEMCC  TNT  node  in  the  JMSM  where 
they  were  manually  pushed  through  a  firewall  to  the  China  Lake  Community  Of  Interest 
(COI)  unclassified  network. 

•  The  display  of  these  events  in  the  CTP  requires  that  the  location  of  the  event  be  included 
in  the  EFT  message.  Because  the  collection  occurred  below  decks  in  the  Jeremiah 


125 


O’Brien  there  was  no  Global  Positioning  System  (GPS)  reeeption  and  the  Crossmateh 
GPS  provided  no  loeation. 

•  The  eoordinates  were  provided  in  VCl  ehat  to  the  BFC  and  manually  inserted  into  the 
CoT  messages. 

•  The  CoT  messages  were  disseminated  by  the  COI  CoT  server  and  sueeessfully  displayed 
in  the  COI  CTP,  passed  through  the  Raytheon  High  Speed  Guard  (RHSG),  and  displayed 
in  the  CFE  CTP. 

Findings 

•  The  EMIO  biometric  event  was  successful.  The  communications  functioned  essentially 
as  planned.  The  use  of  six  Cheetah  radios  to  relay  communication  to  and  from  the 
interior  of  the  ship  was  cumbersome.  Eewer  radios  may  have  successfully  performed  the 
task  but  this  was  not  tested.  The  prime  objective  of  the  EC08  EMIO  event  was  to 
demonstrate  transfer  of  appropriate  data  between  the  various  participating  nodes.  Speed 
of  execution  was  not  a  primary  concern. 

•  The  CEMCC  at  China  Take  had  adequate  Situation  Awareness  (SA)  of  the  biometric 
collection  and  assessment  process  primarily  through  VCl  chat  and  e-mail.  Video  added 
little  because  of  the  very  low  update  rate  and  presented  content  (engine  room).  Audio 
over  VCl  should  be  explored. 

•  The  sending  of  the  biometric  data  from  both  BEC  and  CEMCC  to  ONI  was  redundant. 

•  The  CoT  messages  provided  to  the  CTP  require  amplifying  data  and  only  one  of  the  three 
planned  event  types  was  sent,  but  the  basic  objective  of  displaying  biometric  information 
in  the  COI  and  CPE  CTP  was  successful. 

•  When  GPS  data  are  not  currently  available,  Crossmatch  should  default  to  the  last  good 
GPS  position  to  insert  in  the  EPT  collection  message. 

Assessment  of  EC08  EMIO  Objectives 

Eisted  below  are  the  JBAIIC  EC08  objective  questions  that  relate  to  the  EMIO  event  and  an 
assessment  of  the  degree  to  which  the  objectives  were  satisfied. 

Q-I :  Can  biometric  collection  messages  be  transmitted  from  the  interior  of  a  ship  to  the 
BEC  and  JMSM2?  (Pully  Satisfied) 

The  EPT  formatted  biometric  collection  messages  were  successfully  transmitted  via  Cheetah 
radios  and  the  TNT  network  from  the  interior  of  the  Jeremiah  O’Brien  in  San  Prancisco  to  the 
BEC.  The  biometric  collection  messages  were  posted  to  the  VCl  Pile  Repository  by  the 
collection  team  where  they  were  accessible  to  the  CEMCC  in  the  JMSM2  at  China  Eake.  The 
CEMCC  did  not  have  the  software  required  to  open  EPT  files. 
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Q-2:  Can  biometric  match/no-match  messages  be  transmitted  from  the  BFC  to  the 
boarding  party  in  the  interior  of  the  ship  and  the  JMSM2?  (Fully  Satisfied) 

The  match/no-match  messages  from  the  BFC  were  posted  to  the  VCl  File  repository  where  they 
were  accessible  to  the  boarding  team  and  the  CFMCC  at  JMSM2.  The  match/no  match  results 
were  also  reported  by  the  BFC  in  VCl  chat. 

Q-3:  Can  biometric  detain  messages  be  transmitted  from  ONI/NMIC  to  the  boarding 
party  in  the  interior  of  the  ship  and  the  JMSM2?  (Fully  Satisfied) 

ONI  assessment  messages  were  passed  to  the  CFMCC  in  JMSM2  via  internet  e-mail.  The 
assessments  were  passed  by  the  CFMCC  to  the  boarding  team  via  VCl  chat. 

Q-4:  Were  the  biometric  collection,  match/no-match,  and  detain  events  displayed  in  the 
CTP?  (Partially  Satisfied) 

Biometric  CoT  messages  were  developed  by  the  BFC  and  sent  via  TNT  to  the  CFMCC  TNT 
node  in  the  JMSM  where  they  were  manually  pushed  through  a  firewall  to  the  COI  network  The 
CoT  messages  were  disseminated  by  the  COI  CoT  server  and  successfully  displayed  in  the  COI 
CTP,  passed  through  the  RHSG,  and  displayed  in  the  CFE  CTP.  Only  the  ID  (match/no-match) 
CoT  messages  were  developed  for  the  EMIO  event. 

Q-5;  Did  the  tools  provided  to  the  CEMCC  in  the  JMSM  provide  adequate  SA  to 
exercise  effective  C2  of  biometric  events?  (Eully  Satisfied) 

The  CEMCC  in  the  JMSM  at  China  Eake  had  adequate  SA  of  the  biometric  collection  and 
assessment  process  primarily  through  VCl  chat  and  internet  e-mail.  Video  added  little  because 
of  the  very  low  update  rate  and  presented  content  (engine  room). 


Eessons  Eearned 

•  The  CM  Jump  Kit  with  MOBS  VI. 3. 3  proved  intuitive.  The  operator  had  15  minutes  of 
training.  His  time  capturing  a  full  enrollment  decreased  with  each  enrollment.  By  his  third 
enrollment,  he  required  no  assistance. 

•  The  Scanshell  scanner  is  not  plug-n-play.  The  scanner  had  to  be  configured  at  site.  The  BEC 
TASC  branch  has  performed  limited  testing  on  the  Scanshell. 

•  The  Cross  Match  SeekID  and  the  Cheetah  radios  are  prototype  systems  and  performed  well 
during  this  first  operational  use. 

•  The  file  from  the  Cross  Match  SeekID  was  rejected  at  the  BEC  due  to  incomplete  enrollment 
procedures  during  the  practice  day  at  China  Eake.  During  the  actual  MDA  day,  the  Seek-ID 
files  transmitted  to  the  BEC  from  the  cave  were  at  100%  integrity  and  accepted  by  the  BEC 
database. 
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•  Standards  and  guidelines  should  be  established  for  the  BFC’s  VAMT  to  promote  procedural 
commonality  and  foster  interoperability.  The  tool  should  undergo  a  complete  independent 
verification,  validation  and  accreditation  according  to  DoD  Instruction  5000.62. 
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13.0  Appendix  G:  Trident  Warrior  2008  (TWOS)  Results 


Trident  Warrior  (TW)  is  a  yearly  Navy  operational  experiment  that  focuses  on  Net-Centric 
applications.  TWOS  included  an  MDA  Focus  Area.  Tables  G1  contain  each  of  the  MDA 
objectives  and  measures.  Following  the  tables  are  the  results  for  each  of  the  objectives. 


Each  of  the  TW  experiments  has  a  set  of  Focus  Area,  with  MDA  being  one  for  TWOS.  Each 
Eocus  Area  has  a  set  of  experimentation  Objectives.  Experiment  Threads  are  sets  of  questions, 
associated  metrics,  data  capture,  scenarios,  etc.  The  table  contains  each  of  the  MDA  Objectives, 
their  associated  Questions  to  be  answered,  and  the  Measures  and  Surveys  that  produce  the 
information  to  answer  the  Questions.  Results  are  the  Context  under  which  the  information  was 
gathered.  Question  answer,  and  the  impact  of  the  context  on  the  result. 


Table  18.  TWOS  MDA  Experiment  Threads  objectives  and  results. 


Thread  # 

MDA-01.01 

Objective 

Enable  effective  maritime  domain  awareness  and  analysis  among  distributed  partners. 

Question 

Does  CMA  create  an  accessibie,  accurate  and  usable  user-defined  operationai  picture  in  the  maritime 
environment,  utilizing  muitipie  data  sources  from  multipie  security  ievels? 

Measures 

Which  Chat  used.  Awareness  accuracy. 

Surveys 

Accessible,  Accurate  and  Usable 

Context 

Quest 

Ans 

Accessible:  Yes,  CMA  is  accessibie.  Users  reported  being  abie  to  access  data  and  that  it  reduced 
their  current  workioad. 

Accurate:  Yes,  90%  of  users  either  agreed  or  strongiy  agreed  that  they  could  maintain  awareness 
of  maritime  activity  through  CMA. 

Usable:  Yes,  Users  were  able  to  easily  create  watch  areas  and  do  what  they  need  to  do  for  a 
mission.  It  shouid  be  noted  that  many  users  reported  "N/A"  with  regard  to  ease  of  create  briefing 
materials  and  methods  for  creating  watch  areas. 

Cntxt 

Imp 

Thread  # 

MDA-01.02 

Objective 

Enable  effective  maritime  domain  awareness  and  analysis  among  distributed  partners. 

Question 

Does  CMA  faciiitate  the  reiiable  and  timely  analysis  of  maritime  information  via  information 
exchange  and  anaiytic  toois? 

Measures 

Timeiy,  reiates  to  the  pianning  or  decision-making  process,  dependent  on  ship's  iocation  and 
intentions. 

Surveys 

Reiiabie  and  Timeiy  Use  the  JCTD  instrument?  Suffice  for  TW?  Inciude  on  Web  Surveyor? 

Context 

Observers  were  requested  to  make  observations  that  wouid  provide  data  to  evaluate  the  reliability 
and  timeliness  of  anaiytic  information  provided  by  CMA.  Usage  conditions  did  not  permit  observation 
data  on  reiiabiiity  and  timeiiness  to  be  gathered. 
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Quest  Based  on  feedback  from  surveys,  CMA  assisted  in  facilitated  the  reliable  and  timely  analysis  of 

Ans  maritime  information.  With  regards  to  reliability,  users  reported  that  it  is  important  to  be  able  to 

view  underlying  data  provided  by  CMA  in  evaluating  tracks.  With  regards  to  the  timely  attribute, 
users  reported  that  being  able  to  access  data  from  one  source  (CMA)  was  faster  than  existing 
procedures.  Users  also  stated  that  it  made  their  job  easier  to  be  able  to  correlate  and  store 
Information  in  a  single  system.  Finally,  63%  of  users  strongly  agreed  that  response  time  was 
adequate. 

Cntxt  Data  for  assessment  of  this  thread  is  restricted  to  survey  data. 

Imp _ 


Thread  # 

MDA-01.03 

Objective 

Enable  effective  maritime  domain  awareness  and  analysis  among  distributed  partners. 

Question 

Does  CMA  reliably  and  accurately  detect  when  conditions  of  interest  are  met? 

Measures 

When  algorithm  detects  COI.  Detection  accuracy. 

Surveys 

Reliable,  Accurate,  Relevant 

Context 

Observers  were  requested  to  make  observations  that  would  provide  data  to  evaluate  the  reliability 
and  accuracy  of  CMA's  ability  to  provide  reliable  and  accurate  information  regarding  conditions  of 
interest.  Usage  conditions  did  not  permit  data  on  reliability  and  accuracy  to  be  gathered. 

Quest 

Ans 

Based  on  survey  responses,  CMA  provides  users  with  data  that  is  relevant  and  helpful  to  their 
analysis  of  conditions  of  interest.  No  data  regarding  'reliability'  or  'accuracy'  of  the  data  was 
provided  for  assessment. 

Cntxt 

Imp 

A  complete  assessment  of  this  thread  could  not  be  accomplished  due  to  limited  data. 

Thread  # 

MDA-01.04 

Objective 

Enable  effective  maritime  domain  awareness  and  analysis  among  distributed  partners. 

Question 

Does  CMA  provide  the  flexibility  to  use  pre-defined  conditions  of  interest  and  to  create  new,  relevant 
conditions  of  interest  in  a  timely  manner,  for  the  generation  of  alerts? 

Measures 

Logging  of  conditions  of  interest.  Specify  what  system  conditions  are  needed  for  correlation. 

Surveys 

Yes  --  Flexible,  Manageable,  Relevant,  Timely 

Context 

Observers  were  requested  to  make  observations  that  would  provide  data  to  evaluate  the  degree  to 
which  CMA  is  flexible  and  manageable  in  the  provision  of  relevant  conditions  for  the  generation  of 
alerts.  Usage  conditions  did  not  permit  data  on  reliability  and  accuracy  to  be  gathered. 

Quest 

Ans 

Yes,  the  majority  of  users  reported  liking  the  ability  that  CMA  gives  them  to  leave  an  alert  in  a 
vessel  file  that  can  be  used  by  other  analysts.  Users  also  reported  being  able  to  rely  on  pre-defined 
alerts  to  stay  informed  about  vessels  of  interest.  Regarding  the  timely  attribute,  users  reported  that 
query  functions  saved  significant  time.  Specifically,  it  was  helpful  to  be  able  to  filter  by  location, 
time,  vessel  name  and  vessel  attributes. 

Cntxt 

Imp 

Data  for  assessment  of  this  thread  is  restricted  to  survey  data. 

Thread  # 

MDA-02.01 

Objective 

Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels,  cargo,  personnel  or  MDA  conditions  of 
interest. 

Question 

Can  vessels  be  tracked  accurately  utilizing  the  various  feeds  and  sources  in  MASTER? 
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Measures  would  like  to  gather  BHR  sniffer  data  (but  JWICS)  Network  load  --  bits  per  load  increased  with 
MASTER--the  JTAA  responsibility  --  Persistence. 

Surveys  Efficient,  Any  other  non-ThreadEx  uses  suggested? 

Context 

Quest  Survey  Summary.  MASTER  was  considered  to  be  an  efficient  vessel  tracking  tool  using  various  feeds 

Ans  and  sources,  although  respondents  were  divided  on  whether  MASTER  reduced  their  overall  workload. 

Respondents  agreed  they  could  track  vessels  as  needed,  that  MASTER  is  very  useful  for  maritime 
analysis  (especially  the  track  analysis  and  plotting  features)  and  that  its  capability  to  integrate 
multi-source  intelligence  and  information  with  fused  SuperTrack  data  is  valuable.  Respondents 
mostly  agreed  that  MASTER'S  capability  to  automatically  acquire  data  on  vessels  of  interest  is 
superior  to  current  data  acquisition  processes.  MASTER'S  track  analysis  and  graphing  capabilities 
were  among  its  strengths,  but  its  archive  database  was  often  slow  to  respond  to  queries.  MASTER 
was  easy  to  use  when  creating  new  vessel  tracks.  Its  user  interface  could  be  improved  by  making 
the  vessel  of  interest  category  more  salient  and  improving  the  mapping  program. 

Observer  Log  Summary.  The  vessel  track  information  in  MASTER  was  persistent.  Tracks  remained 
displayed  until  the  operator  cleared  them.  Some  users  questioned  the  completeness  or  accuracy  of 
the  information  in  MASTER'S  database  and  supplemented  it  with  other  sources  of  maritime 
information,  such  as  SeaLink.  Two  users  preferred  ASA  over  MASTER  because  they  judged  it  to  be 
faster  and  easier  to  use,  not  because  it  is  more  accurate  than  MASTER  Various  usability  issues  were 
identified,  especially  with  respect  to  querying  MASTER'S  database  and  defining  alerts. 


Cntxt 

Imp 


Thread  #  MDA-02.02 

Objective  Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels,  cargo,  personnel  or  MDA  conditions  of 
interest. 

Question  Are  the  Pre-defined  alarms  and  alerts  in  MASTER  reliable,  trusted  and  accurate? 

Measures  Accuracy  with  respect  to  alerts  provided  by  MASTER. 

Surveys  Reliable,  Trusted  &  Accurate  (assistance  to  MASTER  for  building  instruments,  responsibility  for 

content  is  JTAA). 

Context 

Quest  Respondents  agreed  that  MASTER'S  pre-defined  alerting  capabilities  are  better  than  current 

Ans  methods.  They  also  agreed  that  they  could  count  on  MASTER'S  pre-defined  alerts  to  detect 

conditions  of  interest.  Respondents  agreed  that  MASTER'S  pre-defined  alerts  are  very  useful  for 
identification  of  potential  maritime  threats.  MASTER  can  provide  named  data  faster  than  current 
systems,  such  as  GALE.  Respondents  were  split  on  whether  MASTER  processes  multiple-source 
intelligence  faster  than  current  methods.  The  main  criticism  of  MASTER  focused  on  its  slowness, 
especially  in  response  to  database  queries;  current  tools  (GALE,  Sealink)  were  described  as  being 
faster. 

Cntxt 

Imp _ 


Thread  # 

MDA-02.03 

Objective 

Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels,  cargo,  personnel  or  MDA  conditions  of 
interest. 

Question 

Are  the  User-defined  alarms  and  alerts  in  MASTER  reliable,  trusted  and  accurate? 
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Measures 

Accurate.  Reliable  &  Accurate  (assistance  to  MASTER  for  building  instruments,  responsibility  for 
content  is  JTAA). 

Surveys 

Reliable,  Trusted  &  Accurate,  (assistance  to  MASTER  for  building  instruments,  responsibility  for 
content  is  JTAA). 

Context 

Quest 

Ans 

Survey  Summary:  Respondents  agreed  that  MASTER'S  user-defined  alerts  supported  accurate 
detection  of  conditions  of  interest.  User-defined  alerts  were  considered  to  be  an  excellent  feature, 
able  to  respond  to  the  dynamic  maritime  threat  environment,  assuming  that  users  provide  accurate 
rules  for  alerts.  Respondents  agreed  that  it  is  easy  to  create  or  modify  user-defined  alerts  in 

MASTER,  noting  that  alert  templates  simplify  this  task.  Several  respondents  stated  that  the  vessel 
tracking  system  they  currently  use  does  not  support  user-defined  alerts. 

Observer  Log  Summary:  Users  could  accurately  create  and  use  user-defined  alerts.  The  polygon 
tripwire  tool  worked  well.  Although  MASTER  was  used  to  successfully  accomplish  vessel  tracking 
tasks,  its  reliability  was  sometimes  at  issue.  For  example,  problems  were  encountered  with  mouse 
control  of  map  zoom  function,  necessitating  a  zoom  out  all  the  way  to  world  view  and  then  re-zoom 
back  to  area  of  interest.  Operators  at  COMPACFLT  used  Firefox  preferentially  over  Internet  Explorer, 
which  was  unreliable  when  attempting  to  run  MASTER. 

Thread  # 

MDA-02.04 

Objective 

Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels  or  MDA  conditions  of  interest. 

Question 

Does  MASTER  allow  for  accurate  and  usable  assembly  of  a  User  Defined  Awareness  Picture  (UDAP) 
in  the  maritime  environment,  utilizing  multiple  data  sources  from  multiple  security  levels  and 
incorporating  Geographic  Information  Systems  (GIS)  of  Infrastructure? 

Measures 

Accurate  (may  not  be  relevant  in  this  sense).  Alerts  may  need  to  be  logged  by  the  system--in 
addition  to  observed  alerts  (captured  by  observer  logs). 

Surveys 

Accurate  &  Usable  Define  responsibilities  for  developing  survey  questions--JTAA/NPS/PSE 

Context 
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Quest  Survey  Summary.  Respondents  agreed  that  they  could  maintain  awareness  of  the  maritime 

Ans  environment  through  the  operational  picture  in  MASTER,  but  they  differed  on  whether  all  the  data 

sources  needed  to  perform  their  tasks  were  available  in  MASTER.  Sealink  was  suggested  as  a  data 
source  that  could  be  integrated  with  MASTER.  MASTER'S  strengths  include  its  ability  to  support 
information  sharing  among  users,  receive  intelligence  from  sources  at  the  SECRET  level  and  below, 
create  watch  areas  and  briefing  materials,  and  define  the  operational  picture  to  maintain  awareness 
of  the  maritime  environment.  MASTER'S  user  interface  was  judged  to  be  clear,  consistent,  and  easy 
to  understand,  and  most  respondents  found  it  easy  to  get  MASTER  to  do  what  they  wanted,  given 
adequate  training  and  practice,  which  are  needed  to  reach  proficiency.  Usability  drawbacks  include 
multiple  click  controls  (2  or  3  clicks)  for  various  features  and  slow  response  times,  especially  to 
database  queries.  MASTER'S  display  also  has  a  tendency  to  become  too  cluttered  for  effective 
viewing  under  higher  traffic  conditions. 

Observer  Log  Summary.  System  connectivity  was  usually  adequate  during  the  MASTER  scenarios, 
although  the  system  would  go  down  at  times,  preventing  the  execution  of  the  scenarios.  Delays  in 
MASTER  performance  seemed  most  likely  due  to  MASTER  server  performance.  Multiple  simultaneous 
searches  also  slowed  system  response  time.  Such  response  time  delays  could  hinder  the  timely 
assembly  of  a  UDAP  in  up-tempo  operations.  The  accuracy  of  the  results  generated  by  MASTER 
depend  on  a  user's  skill  in  selecting  and  applying  the  appropriate  filtering  criteria.  The  need  to 
manually  correlate  some  data  elements  across  various  MASTER  screens  introduces  the  potential  for 
user  error.  Correlations  should  be  system  automated.  Numerous  usability  issues  were  observed  that 
could  prevent  the  generation  of  accurate  and  usable  UDAPs,  including:  the  inability  to  cancel  a 
query;  the  use  of  non-standard  command  input  (e.g.,  2-  or  3-click  inputs  and  the  map's  zoom 
controls);  the  inability  to  perform  multiple  simultaneous  queries;  the  inability  to  sort  search  results 
by  the  result's  fields;  and  the  inability  to  perform  follow-on  searches  based  on  a  prior  search.  Also, 
the  addition  of  visual/auditory  cues  for  the  alarm/alert  windows  would  prompt  users  to  review  them 
for  status  changes,  which  several  users  failed  to  do. 


Thread  #  MDA-02.05 

Objective  Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels,  cargo,  personnel  or  MDA  conditions  of 
interest. 

Question  Does  MASTER  allow  for  ingest,  accurate  vetting  and  display  of  people  data  associated  with  maritime 
vessels  and  maintain  an  association  of  certain  people  with  ships  both  past  and  present? 

Measures  Accurate.  What  system  logs,  and  how  will  they  be  extracted? 

Surveys  Accurate 

Context 

Quest  This  Thread  was  dropped  -  People  Awareness  feature  is  not  implemented  in  the  current  version  of 

Ans  MASTER 

Cntxt 

Imp _ 


Thread  # 

MDA-02.06 

Objective 

Evaluate  the  capability  to  track  vessels  using  National  Technical  Means  and  Open  Source  Data  and 
provide  operators  with  alerts  as  related  to  suspect  vessels,  cargo,  personnel  or  MDA  conditions  of 
interest. 

Question 

Does  MASTER  provide  a  useful  and  accurate  replication  of  ONI's  Global  Trader  cargo  tracking  and 
alerting  capability,  allowing  a  linkage  of  vessels  and  related  cargo? 

Measures 


Accurate 
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Surveys  Accurate  &  Usable 

Context 

Quest  Survey  Summary.  Most  respondents  (13  of  20)  agreed  that  MASTER  allowed  them  to  determine 

Ans  linkages  between  cargo  and  vessel.  Several  respondents  noted  drawbacks  in  the  content  and 

capabilities  of  the  Cargo  database,  such  as  cargo  data  being  limited  and  hard  to  find  in  MASTER,  but 
others  said  it  exceeded  their  expectations. 

Observer  Log  Summary.  Cargo  information  provided  by  MASTER  was  usable,  but  could  be  more 
complete.  Cargo  Link  was  preferentially  used  Instead  of  MASTER  to  obtain  cargo  information  by 
several  users,  who  judged  Cargo  Link  to  be  more  extensive.  According  to  an  observer,  the  MASTER 
Cargo  results  screen  has  redundant  information  and  no  Interactivity.  It  requires  users  to  review 
pages  of  data,  document  potential  leads  into  a  list  of  items,  and  to  then  run  many  individual  queries. 
Some  users  indicated  they  need  the  ability  to  select  Individual  cargo  results  and  draw  the  tracks  on 
the  map  in  a  manner  similar  to  the  track  query  select/draw  function  The  accuracy  of  MASTER'S 
database  appeared  to  be  adequate,  although  its  completeness  compared  to  a  specialized  cargo 
database,  such  as  Cargo  Link,  was  questioned. 


Thread  #  MDA-04.01 

Objective  Assess  the  usability  and  operational  utility  of  PANDA  to  intelligence  analysts  and  watchstanders  for 
representing  normal/predicted  vessel  behavior  and  detecting/alerting  deviations  from  normal  vessel 
behavior 

Question  Is  PANDA'S  representation  of  predicted/normal  vessel  behavior  and  detection  of/alerting  to 

deviations  from  normal  relevant/clear  to  and  trusted  by  Intelligence  analysts  and  watchstanders? 

Measures 

Surveys  Usable  -  Clear,  Relevant  and  Trusted:  Users  can  be  analysts  and  operators  (different  tasks  / 

different  perspective,  assessment  of  utility,  etc)  Survey  or  interviews  for  depth.  Observers  verify 
that  PANDA  alerts  deviation  when  it  occurs  (based  on  scenario  events)  Clear:  Determine  whether 
users  notice  and/or  use  alert 

Context 

Quest 

Ans _ 

Cntxt 

Imp _ 

Thread  #  MDA-04.02 

Objective  Assess  the  usability  and  operational  utility  of  PANDA  to  intelligence  analysts  and  watchstanders  for 
representing  normal/predicted  vessel  behavior  and  detecting/alerting  deviations  from  normal  vessel 
behavior. 

Question  Does  PANDA  provide  a  needed  capability  for  monitoring  and  providing  situation  awareness  about 
vessels  of  interest  and  is  the  information  provided  by  PANDA  applicable  Information  to  support  the 
analysis  of  VOIs? 

Measures 

Surveys  Utility:  Needed,  Applicable  In  addition,  questions  to  operators,  as  well  as  ONI  watchfloor  operator  to 
help  capture  if  PANDA  information  gets  into  decision  chain 

Context 

Quest 

Ans _ 

Cntxt 

Imp _ 
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Thread  # 


MDA-06.01 


Objective  Integrate  unclassified  track  and  reference  data  from  DoD,  coalition,  and  commercial  sources  to 
provide  the  unclassified  COP  portion  of  Global  Maritime  Situational  Awareness. 

Question  Can  various  track  data  sources  be  integrated  into  MIDAS  in  a  timely  manner  to  provide  a  relevant 
picture  in  support  of  Global  Maritime  Situational  Awareness? 

Measures 

Surveys  timely  and  relevant 

Context  Limited  Observation  data  was  collected. 

Quest  MIDAS  does  provide  the  ability  to  provide  a  relevant,  unclassified  picture  in  support  of  Global 

Ans  Maritime  Situational  Awareness  in  a  relatively  timely  manner.  The  observer  offered  some  design 

suggestions  that  should  improve  the  operator's  ability  to  make  modifications  in  less  time  --  i.e., 
allow  changes  to  be  made  at  both  line  and  pie  charts;  have  ability  to  filter  on  vessel  size;  auto-insert 
the  lat/long  value  into  the  data  box. 

Cntxt  Data  are  mostly  operator  estimates  versus  actual  experience  with  MIDAS  in  an  operational  setting. 

Imp 


Thread  #  MDA-06.02 

Objective  Integrate  unclassified  track  and  reference  data  from  DoD,  coalition,  and  commercial  sources  to 
provide  the  unclassified  COP  portion  of  Global  Maritime  Situational  Awareness. 

Question  Does  MIDAS  provide  relevant  and  timely  information  to  distributed  planners? 

Measures 

Surveys  timely  &  relevant 

Context  There  was  only  one  instance  of  MIDAS  in  TWOS;  therefore,  there  was  no  ability  to  execute 
distributed  planning. 

Quest  MIDAS  was  estimated  to  have  the  ability  to  provide  relevant  unclassified  information  with  other 

Ans  agencies  and  non-traditional  partners.  Estimates  of  timeliness  were  divided;  determination  of 

timeliness  requires  actual  execution. 

Cntxt  Reports  represent  participant  estimates  only,  as  no  distributed  planning  was  actually  attempted. 

Imp 


Thread  # 

MDA-06.03 

Objective 

Integrate  unclassified  track  and  reference  data  from  DoD,  coalition,  and  commercial  sources  to 
provide  the  unclassified  COP  portion  of  Global  Maritime  Situational  Awareness. 

Question 

Can  MIDAS  be  trusted  to  exchange  relevant  and  reliable  data  and  alerts  with  MDA  Data  Sharing  COI, 
Google  Apps,  NCIS,  FIAC,  and  also  EMIO,  VBSS,  TBCMS  and  MCSBEN  (boarding  data)? 

Measures 

reliable  &  relevant  Data  logs  from  MIDAS  and  SIDI. 

Surveys 

reliable  &  relevant 

Context 

Limited  observation  data. 

Quest 

Ans 

In  general,  operators  believed  that  MIDAS  could  assist  them  to  exchange  relevant  and  reliable  data 
with  multiple  sources.  Actual  data  exchange  occurred  successfully  with  MDA  DS  COI. 

Cntxt 

Imp 

The  assessment  of  reliability  of  data  provided  via  MDA  DS  COI  is  based  on  minimal  observances; 
therefore,  actual  reliability  over  multiple  times  and  situations  is  not  known. 

Thread  #  MDA-10.01 
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Objective 

Develop  an  effective  SOA  Data  Exchange  and  Correlation  Measure  network  traffic  flow  and  Secure 

Data  Exchanges  and  Cross  Domain  Communications. 

Question 

Can  SIDI  provide  a  flexible,  secure  and  timely  SOA  web  service  translation  capability  between 
unclassified  MDA  data  sources? 

Measures 

Timely  info  exchange,  secure  decryption 

Surveys 

Report  of  encryption  -  decryption  test  for  secure.  Timely  information  exchanged  (by  packet  size, 
type,  location).  Report  of  ingestion  of  multiple  formats  and  conversion  appropriate  to  customers  for 
flexible 

Context 

0  There  were  3  SIDI  servers  for  scenario  1  -  on  EMIO  laptop  atop  COMPOSE  3.0  of  USS  Comstock, 
shore-based  (Pt.  Loma)  on  CANES  (ISNS-NGT)  blade  server,  and  a  reachback  server  on  a  Windows 
Server  at  St.  Louis,  MO. 

0  There  were  2  SIDI  servers  for  scenario  2  -  on  VPN  server  hosted  at  St.  Louis,  MO,  and  the 
reachback  server  on  a  Windows  Server  at  St.  Louis,  MO. 

Quest 

Ans 

SIDI  was  able  to  provide  a  flexible  and  timely  means  of  translating  and  transmitting  data  to  many 
receivers.  Due  to  restrictions,  the  secure  component  of  the  experiment  was  discarded. 

Cntxt 

Imp 

Performance  issues  on  VMWare  servers  resulted  in  the  encryption  -  decryption  test  for  secure  being 
removed  from  the  experiment. 

Thread  # 

MDA-11.01 

Objective 

The  MDA  DS  COI  provides  a  seamless  information  sharing  infrastructure  to  provide  data  producers 
and  consumers  a  single/common  methodology  for  exposing,  discovering,  publishing  and  subscribing 
to  unclassified  MDA  data. 

Question 

Is  the  MDA  DS  COI  AIS  data  available  and  accessible  in  a  timely  manner  to  the  unanticipated  user? 

Measures 

Available:  information  flow  of  AIS  data.  Time  required  to  complete  view  VOLPE  unclassified  MDA 
data  (VOPE,  Global  Hawk,  SMS/SCC  J,  augmented  unclassified  MDA,  historical  unclassified  MDA, 
anomalies  for  unclassified  MDA). 

Surveys 

Accessible  -  Could  the  operator  access  the  MDA  DS  COI  GMMS  Website?,  Were  the  operator's 
information  sharing  attempts  seamless  (i.e.  did  the  site  provide  a  single  source  to  access 
unclassified  MDA  data  from  multiple  sources)? 

Timely  -  Was  data  available  when  needed  through  MDA  DS  COI  GMMS  website? 

Available  -  view  of  MDA  data  (Navy  unclassified,  ARMS  unclassified.  Global  Hawk  unclassified, 
SMS/SCC  J  unclassified,  augmented  unclassified,  historical  unclassified,  anomalies  for  unclassified. 

Usable  -  Did  the  operator  appear  overwhelmed/confused/make  observable  errors  at  any  point  while 
using  MDA  DS  COI  GMMS  website? 

Context 

Observations  took  place  at  COMTHIRDFLEET  and  COMPACFLEET,  during  discreet  usage  events 
between  0800  and  1530  PDT. 

Quest 

Ans 

Overall,  MDA  DS  COI  (web  version)  was  accessible  by  all  operators.  However,  there  were  some 
challenges  with  regard  to  timeliness;  operators  stated  that  the  website  was  slow.  The  majority  of 
data  sources  (5  of  8)  were  available  through  MDA  DS  COI.  Survey  respondents  reported  moderate 
availability  of  data. 

Cntxt 

Imp 

From  Qualitative  data:  The  timing  of  usage  events  may  have  resulted  in  non-representative 
observations  of  availability  of  data  sources,  due  to  time  zone  differences. 

Thread  # 

MDA-11.02 

Objective 

The  MDA  DS  COI  provides  a  seamless  information  sharing  infrastructure  to  provide  data  producers 
and  consumers  a  single/common  methodology  for  exposing,  discovering,  publishing  and  subcribing 
to  unclassified  MDA  data. 

Question 

Is  the  MDA  DS  COI  data  available  to  the  anticipated  user  within  a  reasonable  timeframe? 

Measures 

Yes  --  Timely,  Available:  AIS  information  flow 
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Surveys  Yes  --  Timely,  Available,  Usable 

Context  Links  to  MDA  DS  COI  data  were  embedded  in  four  different  TWOS  technologies  -  CLA,  MASTER, 

MIDAS,  and  CMA.  It  was  incumbent  upon  Technical  SMEs  and/or  observers  to  ensure  that  operators 
(a)  understood  the  relationship  between  the  MDA  DS  COI  and  the  host  technology,  and  (b) 
attempted  to  use  MDA  DS  COI.  This  did  not  always  occur;  in  the  cases  of  CLA  and  CMA,  no  usable 
data  was  reported. 

Quest  •  Accessible:  Yes.  Qualitative  feedback  from  observer  logs  for  MIDAS  and  MASTER  technologies 

Ans  indicated  that  users  were  able  to  access  MDA  data  from  various  sources  through  the  MDA  DS  COI. 

•  Timely:  No.  Feedback  from  observer  logs  for  MASTER  indicated  that  operators  had  difficulty 
staying  on  mission  timelines  due  to  challenges  with  running  queries.  Specifically,  some  operators 
took  notes  by  hand,  others  reported  difficulty  passing  information  from  one  query  to  the  next, 
others  reported  only  being  able  to  do  1  query  at  a  time. 

•  Usable:  Feedback  indicates  that  Usability  was  relatively  low.  Users  were  able  to  identify  suspect 
VOIs.  However,  it  was  difficult.  One  observer  stated  "This  user  interface  needs  major  improvement 
with  respect  to  correlating  data  relationships  between  popup  windows  and  the  map,  and  the  track 
data  playback  function.  The  process  requires  the  operator  to  bounce  between  track  data  popup 
spreadsheets  and  layer  tools  lists.  The  operators  had  to  manually  document  record  track  identifier 
numbers  to  compare  data  list  on  other  windows.  This  process  is  prone  to  errors  as  demonstrated 
during  this  event." 


Cntxt  Assessments  with  all  planned  technologies  did  not  occur. 

Imp  Given  the  hosted  relationship  of  MDA  DS  COI  for  this  event,  GUI  design  was  rudimentary,  as 

reflected  in  feedback  regarding  Usability. 


Thread  # 

MDA-12.01 

Objective 

Determine  if  FP  Portal  is  a  viable  collaboration  platform  to  exchange  Maritime  information  with 
mission  partners. 

Question 

Is  the  FP  Portal  accessible,  reliable  and  usable  for  the  sharing  of  relevant  information  between 
distributed  NCIS  agents? 

Measures 

accessible  (log  on  success  and  failure),  reliable  (up  and  down  times):  consider  bandwidth  usage 
measurements  (FP  servers.  BW  measures  at  BHR.  Different  nodes  may  have  different  means 
(expeditionary  comms,  PDA,  PC  etc)  to  log  into  the  Portal. 

Surveys 

operator  experience  (accessibility,  reliability,  usability).  HI,  Bahrain,  Dubai,  Singapore,  NCIS 
NORTHWEST,  C3F,  NCIS  HQ  at  Navy  Yard  DC.  Possibly  on  BHR.  PACFLT  for  Spiral-1?  Relevant 

Context 

NCIS  Agents  and  Field  Officers  at  several  remote  locations  logged  into  FP  Portal  and  participated  in 
collaboration  experiment  to  'solve'  a  case. 

Quest 

Ans 

Accessibility  of  the  FP  Portal  was  high,  even  from  mobile  device.  Information  entered  into  the  Portal 
was  reliably  saved  and  available.  Connectivity  was  good,  aithough  some  problems  were  reported  via 
access  with  NMCI.  Most  users  reported  that  features  of  the  Portal  were  easy  to  understand  and 
customize,  although  one  user  requested  a  simpler  interface.  It  was  suggested  that  the  Chat  feature 
of  the  Portal  may  not  be  the  best  method  to  Communicate  urgent  messages  as  it  was  often  unclear 
whether  desired  contacts  were  logged  in;  this  would  require  log-in  and  usage  procedures  to  be 
adopted. 

Mimicked  real-world  usage  of  the  portal. 

Thread  # 

MDA-13.01 

Objective 

Demonstrate  the  ability  to  share  Law  Enforcement  information  derived  from  LInX  and  make 
associations  with  other  disparate  information  provided  via  MDA. 

Question 

Does  LInX  provide  an  accessible,  reliable  and  usable  information  environment  for  assisting  in  making 
connections  between  other  information  types,  such  as  from  other  MDA  sources? 

Measures 

accessible,  reliable:  system  log  (from  LInX) 
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Surveys  operator  perspective  of  accessibility,  reliability,  and  usability  --  also  utility  of  the  available  info  to  the 
MDA  community  (all  access  to  LInX  via  FP  Portal  for  TW).  Observations  to  be  used  to  report  means 
of  access  (Expeditionary  comms  device,  TREO,  Blackberry,  workstation  etc)  and  performance 
(access,  reliability  and  usability). 

Context  One  respondent  reported  not  using  LInX  but  requesting  it  via  the  MEFO. 

Quest  LInX  was  accessible  via  the  FP  Portal.  Participants  reported  excellent  ability  to  move  between  FP 

Ans  Portal  and  LInX.  The  information  available  through  LInX  was  rated  as  Moderate  to  Fligh  value, 

although  one  respondent  wanted  more  depth  of  social  linkage  information.  Respondents  generally 
trusted  the  sources  that  provided  information  to  LInX. 

Cntxt  One  respondent's  replies  do  not  reflect  use  of  LInX.  This  respondent's  experience  to  gain  access  to 

Imp  LInX  is  likely  a  function  of  the  experiment  control. 


Thread  # 

MDA-14.01 

Objective 

Expand  the  NCIS  source  infrastructure  to  include  contacts  in  pillar  MDA  domain  areas:  cargo, 
people,  critical  infrastructure,  vessels. 

Question 

Can  field  agents  establish  sources  across  the  MDA  pillars  of  vessels,  cargo,  people,  and  critical 
infrastructure? 

Measures 

field  agent  self-report  of  numbers  and  descriptions  of  sources  that  were  established  (include 
associated  MDA  pillar  area  of  each  source) 

Surveys 

able  to  re-establish  with  contacts  on  demand?  (reliable)  how  much  human  resources  involved  In 
creating  and  maintaining?  (workload=efficient) 

Context 

Only  one  location  provided  Observation  data.  Reports  suggest  that  this  Objective-Question  was  not 
fully  tested. 

Quest 

Ans 

The  amount  of  information  exchanged  did  not  increase  appreciably  during  the  experiment  over  real- 
world  conditions;  the  report  of  contacts  established  was  not  hiah.  It  was  estimated  that  workload  to 
establish  and  maintain  contacts  would  be  the  same  or  higher,  but  that  re-connecting  with  those 
contacts,  once  established,  would  be  easy.  There  was  strong  agreement  that  the  MDA  Social 

Network  is  potentially  a  very  valuable  source  of  contacts. 

Cntxt 

Imp 

Responses  are  largely  estimations  vs.  actual  experience.  This  may  have  been  a  function  of  the 
scenario  or  the  fact  that  participants  had  real-world  responsibilities  in  addition  to  participating  in  the 
experiment. 

Thread  #  MDA-14.02 

Objective  Improved  C2  processes  among  Navy,  International,  Industry,  Non-Governmental,  and  Interagency 
organizations  through  NCIS,  to  enable  a  Law  Enforcement  Network  to  support  global  Maritime 
Domain  Awareness  activities  so  that  Operational  and  Tactical  level  decision  makers  recognize  the 
value  of  law  enforcement  activities  in  support  of  critical  MDA  processes. 

Question  Are  TWOS  MDA  net-centric  services  accessible,  reliable,  capable  and  usable  as  integrated  features 
for  nodes  within  the  MDA  Community  of  Interest? 

Measures 

Surveys  Need  to  define  the  integration  of  services  that  should  be  available  at  nodes.  Accessible,  Reliable, 
Capable,  Usable 

Context  Only  1  location  provided  Observation  data,  the  only  net-centric  services  reported  were  FP  Portal  and 
LInX.  Participants  may  not  have  understood  or  had  time  to  expiore  other  potential  services. 

Quest  The  net-centric  services  reported  (i.e.,  FP  Portal  and  LInX)  were  accessible,  reliable,  and  capable. 

Ans  Responses  at  Objectives  12  and  13  suggest  a  high  level  of  usability,  although  integration  of  services 

was  not  specifically  reported. 

Cntxt  This  Objective-Question  may  not  have  been  fully  tested,  as  FP  Portal  and  LInX  were  only  services 

Imp  reported. 
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Measures 

Surveys 

Context 

Connectivity  was  adequate  during  ~85%  of  the  usage  situations.  However,  there  were  8  reports  of 
long  processing  times  (2-5  min.)  or  timing  out  during  queries. 

Usage  scenario  was  very  directed  -  very  little  free  play. 

Quest 

Ans 

Based  on  survey  and  observer  log  results,  CargoLink  was  a  usable  and  sufficient  data  query  tool  for 
analysts  who  are  not  cargo  experts.  Survey  respondents  indicated  that  they  understood  the  features 
of  CargoLink,  could  easily  manage  data  in  CargoLink,  and  CargoLink  was  adequate  to  perform 
analysis  of  cargo  that  is  associated  with  a  vessel.  However,  roughly  1/3  of  respondents  did  not 
endorse  (i.e.,  they  disagreed  or  were  neutral)  the  relevance  of  CargoLink  to  the  analysis  that  they 
do  for  their  jobs,  and  roughly  1/3  of  operators  were  reported  to  be  frustrated  at  some  point  during 
their  session. 

Cntxt 

Imp 

It  was  not  possible  to  determine  if  connectivity  problems  contributed  to  long  search  times  and  time¬ 
outs. 

With  limited  free  play,  ability  of  the  operators  (who  were  not  cargo  experts)  to  assess  relevance  to 
their  jobs  may  have  been  restricted. 
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14.0  Appendix  H:  Assessment  Framework 

Portions  of  the  following  material  are  presented  in  Seetion  2.  The  full  strueture  is  presented  here 
for  eompleteness. 

14. 1  Assessment  Areas  and  Objeetives 

The  assessment  strueture  is  organized  around  five  Assessment  Areas,  shown  in  Table  II.  The 
basie  MDA  assessment  objectives  are  to  determine 

•  Effeetiveness  and 

•  Military  Utility. 

Definitions  of  both  are  given  in  Seetion  1.4.  Effeetiveness  is  determined  for  all  four  Assessment 
Areas.  Military  Utility  is  determined  only  for  systems. 

MDA  assessment  objectives  are  to  determine  the  following. 

System  Performance  evaluates: 

•  How  well  the  system  performs  its  functions 

•  How  well  the  system  supports  MDA  operations 

•  Warfighter  acceptance  of  the  system 

•  Whether  or  not  the  system  provides  automated  support 

•  Quality  of  system  management  and  security 

Operations  Performance  evaluates: 

•  Quality  of  knowledge  processes,  particularly  in  support  of  MDA 

•  Quality  of  ISR  operations  in  support  of  MDA 

•  Quality  of  MIO  operations 

Warfighter  Performance  focuses  on: 

•  Operator  acceptance  and  understanding  of  the  MDA  mission 

•  Capabilities  of  units  to  carry  out  MDA  operations  and  activities 

•  Human  capabilities  to  carry  out  MDA  tasks 

Organization/Guidance  focuses  on: 

•  MDA  organization  compatibility  other  organizations  and  their  processes 

•  MDA  organization  compatibility  with  MHQ/MOC 

•  Sufficiency  of  agreements  to  enable  MDA  operations 

•  Guidance  sufficiency  to  support  operations 

Supportability  and  Readiness  refers  specifically  to  MDA  Spiral- 1  systems  and  is  reported  by 
PEOC4I.  A  brief  summary  of  their  results  is  provided  here  for  completeness  of  this  report. 
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Table  II  provides  more  granularity  to  the  assessment  strueture,  showing  the  three-level 
evaluation  strueture. 


Table  19,  MDA  Assessment  Areas  and  Structures. 


System  Performance 

Operations  Performance 

Technical  Performance 

Knowledge  Processes 

Information  Retrieval 

Information  Retrieval 

Information  Processing 

Vol  Development 

Information  Sharing 

Vol  Tracking 

Operator  Configurable 

Information  Sharing 

Interoperability 

ISR 

Operations  Support 

Planning 

System  Utility 

Execution 

Standards  and  Guidelines 

PED 

Warfighter  Acceptance 

MIO 

System  Utility 

Planning 

Human-System  Interaction 

Execution 

System  Usage 

Assessment 

System  Training 

Automation 

Organization/Guidance 

Alerts 

MDA  Compatibility 

Information  Processing 

Organization  Alignment 

Smart  Pull 

Process  Alignment 

System  Management  and  Security 

MHQ/MOC  Compatibility 

Organization  Alignment 

Warfighter  Performance 

Process  Alignment 

MDA  Mission 

Agreements 

Mission  Understanding 

Information  Sharing 

Mission  Acceptance 

Shared  Operations 

Unit  Performance 

Guidance 

Manning 

CONOPS 

Activities 

TTP/SOP 

Training 

Standing  Orders 

Human  Performance 

Tasks 

System  Supportability  and  Readiness 

Training 

PEO  Provided 

14.2  Assessment  Metries 

The  overarehing  purpose  of  the  assessment  is  to  determine  Capability  Indieators.  As  noted 
above,  the  basie  indieators  are  Effeetiveness  and  Military  Utility.  These  indieators  are  evaluated 
through  their  ineluded  measures,  MOU,  MOE,  and  MOP.  The  measures  used  for  the  MDA 
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Assessment  Areas  are  shown  in  Table  21,  using  their  attributes.  Attributes  rather  than  measures 
are  shown  in  the  table  for  semantic  ease. 

Table  20,  MDA  Capability  Indicators  (MOE  and  MOU) 


Area 

Effective 

Utility 

Area 

Effective 

Systems 

Accessible 

Capable 

Reliable 

Usable 

Improved 

Needed 

Applicable 

Wanted 

Warfighter 

Capable 

Reliable 

Operations 

Accessible 

Capable 

Reliable 

Organization 

/Guidance 

Accessible 

Capable 

Usable 

These  attributes  are  chosen  from  complete  structure  which  was  developed  for 
NAVNETWARCOM  for  use  in  its  Capabilities  Based  Analyses,  a  component  of  the  JCIDS 
process.  This  structure  was  developed  to  conform  with  three  of  the  JCIDS  JFAs;  NCO,  BA,  and 
C2.  Table  13  lists  the  effectiveness  and  utility  attributes  (which  defined  the  four  MOE  and  four 
MOU).  Under  each  of  the  MOE  attributes  are  their  associated  MOP  attributes. 


Table  21,  Complete  Attribute  Structure, 


Effective 

Accessible 

Reliable 

CaDable 

Usable 

MOE 

Capacity 

Available 

Compatible 

Extensive 

Efficient 

Robust 

Persistent 

Secure 

Assured 

Sufficient 

Flexible 

Accurate 

Timely 

Reach 

Automatic 

Clear 

Trusted 

Manageable 

Relevant 

Compliant 

Deployable 

MOP 

It 

II 

II 

II 

M 

Military  Utility 

Improved 

Needed 

Applicable 

Wanted 

MOU 

Effective 

Utility 

Life-Cfcle 

Personnel 

MOR 


Table  13  shows  that  MOP  are  assigned  to  specific  MOE.  This  shows  principal  assignments,  the 
MOPs  that  would  normally  be  rolled  up  to  provide  an  MOE  determination.  It  is  possible,  and 
not  unusual,  to  use  an  MOP  for  a  different  MOE  than  is  shown  in  the  table,  e.g.,  the  MOP 
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compatible  could  be  used  with  the  MOE  capable  to  determine  if  organizations’  processes  were 
compatible  so  personnel  could  collaborate  over  decision-making. 

14.3  Assessment  Measures 

Table  14  presents  the  measures  that  are  to  be  evaluated  for  each  Assessment  Area.  Attributes  are 
in  bold,  followed  by  their  associated  measures. 

Table  22,  MDA  Assessment  Metrics, 


MDA  Assessment  Metrics 

MOE 

MOP  or  MOU 

Spiral-1  System  Performance  (each  system) 

Technical  Performance 

Improved:  -5  to  +5  rating  of  improvement  over  existing  systems,  by  system 
aspect. 

Needed:  system  fiiis  a  gap  in  existing  capabiiities,  Y/N. 

Applicable:  system  is  appiicabie  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  avaiiabie. 

Information  Accessible:  roll-up  of  information  accessibility. 

Retrieval  Available:  %  of  time  information  is  available. 

Efficient:  number  of  steps  to  access  information. 

Capable:  roll-up  of  capability  to  retrieve  required  information. 

Sufficient:  %  of  information  needed  for  assessment. 

Timely:  time  required  to  retrieve  information. 

Reliable:  roll-up  of  ability  to  obtain  correct  information  when  needed. 

Assured:  information  source  is  identified,  Y/N. 

Robust:  automatic  failover  during  system  problems,  Y/N;  database  backup, 
Y/N. 

Persistent:  %  of  time  down  due  to  system  failure. 

Usable:  roll-up  of  information  usability  for  assessment  and  decision-making. 

Clear:  1-5  rating  of  information  clarity;  1-5  rating  of  GUI  presentation. 

Trusted:  1-5  rating  of  confidence  in  information. 

Information  Capable:  roll-up  of  ability  to  process  ship  and  Vol  information. 

Processing  Available:  information  processing  capabilities,  Y/N;  list  capabilities. 

Efficient:  use  of  information  processing  capabilities,  by  capability,  Y/N. 
Sufficient:  information  processing  capabilities  for  operations  needs,  Y/N. 
Automatic:  automatic  information  processing  available?  Y/N 

Information  Accessible:  roll-up  of  information  sharing  accessibility. 

Sharing  Compatible:  M2M  interoperability,  by  system,  Y/N. 

Available:  %  of  time  information  sharing  available. 

Efficient:  information  sharing  efficient,  Y/N;  number  of  steps  required  to 
share  information. 

Capable:  collaboration  capabilities  provided,  Y/N;  roll-up  of  capability  to  share 
information  with  other  units. 
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Reach:  number  of  units  with  which  information  can  be  shared;  number  of 
units  per  collaboration  session. 

Sufficient:  %  of  required  units  with  which  information  can  be  shared;  %  of 
required  information  that  can  be  shared. 

Timely:  time  required  to  exchange  information. 

Operator 

Configurable 

Capable:  1-5  rating  of  the  ability  of  operator  to  configure  the  system  as  desired. 

Flexible:  Operator  can  configure  information  search,  information 
presentation,  Y/N. 

Sufficient:  fraction  of  required  profile  types  that  can  be  developed. 

Reach:  Number  of  profiles  that  can  be  saved. 

Efficient:  1-5  rating  of  system  configuration  efficiency;  number  of  steps 
required  to  configure  system,  by  configuration  type. 

Interoperability  Compatible:  M2M  system  interoperability,  by  system,  Y/N; 

information  formats  compatibility,  by  system,  Y/N. 


Operations  Support 

System  Utility  Improved:  -5  to  +5  rating  of  improvement  ofMDA  operation  activities  over 
existing  systems,  by  activity. 

Needed:  system  fitts  a  gap  in  existing  support  to  MDA  operations,  Y/N. 
Applicable:  system  is  appiicabie  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operations  center  desire  to  have  system  avaiiabie. 


Standards  and  Usable:  roll-up  assessment  of  Standards  and  Guidelines  for  system  usage 


Guidelines 

Sufficient:  %  of  system  operations  covered  by  guidelines;  standards  to  cover 
information  formats,  Y/N;  standards  to  cover  M2M  interactions,  Y/N. 

Clear:  guidelines  to  direct  system  operation,  Y/N. 

Prepare  a  list  of  those  situations  for  which  Standards/Guidelines  are 
inadequate. 

Warfighter  Acceptance 

System  Utility  Improved:  -5  to  +5  rating  of  improvement  of  operator's  task  performance 


over  existing  systems,  by  task. 

Needed:  system  fitts  a  gap  in  existing  support  to  task  performance,  Y/N. 
Applicable:  system  is  appiicabie  to  MDA  tasks,  by  task,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  avaiiabie. 

Human- 

System 

Interaction 

Usable:  1-5  scale  roll-up  of  human-system-interaction. 

Clear:  1-5  scale  on  GUI  presentation. 

Manageable:  GUI  can  be  configured  to  operator  desired  presentation,  Y/N. 
Relevant:  information  presented  is  relevant  to  operator  task  performance, 

Y/N. 

Timely:  GUI  layout  and  information  presentation  facilitates  rapid  retrieval  of 
needed  information,  Y/N;  time  to  retrieve  needed  information. 

Efficient:  number  of  steps  required  to  retrieve  needed  information;  1-5  scale 
on  GUI  facilitation  of  information  retrieval  efficiency. 

System  Usage  Prepare  a  table  of  frequency  of  Spiral-1  system  use,  by  situation  and  by  task 


System 

Training 

Usable:  1-5  scale  roll-up  of  training  quality. 

Clear:  1-5  scale  on  training  clarity. 

Sufficient:  1-5  scale  on  sufficiency  of  training  to  prepare  operator  for  tasks. 
Relevant:  1-5  scale  on  whether  training  is  relevant  to  mission  and  workflow. 

Automation 

Alerts 

Capable:  alerts  are  provided,  Y/N;  roll-up  of  quality  of  alerts. 
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Automatic: 

Flexible:  alerts  operator  configurable,  Y/N;  number  of  different  types  of  alerts 
available. 

Trusted:  1-5  rating  of  alerts  eliminating  need  to  monitor  situation. 

Sufficient:  %  of  needed  alerts  provided  by  system,  %  of  needed  alert  types 
provided  by  system. 

Timely:  alerts  provided  in  time  to  take  needed  actions,  Y/N. 

Efficient:  1-5  rating  of  efficiency  setting  up  alerts;  number  of  steps  required 
to  set  up  alerts. 

Information  Capable:  machine  assisted  information  processing  provided,  Y/N;  system 

Processing  automated  information  processing  available,  Y/N;  roll-up  assessment  of 

information  processing  capability. 

Automatic:  hands-off  information  processing  available,  Y/N. 

Flexible:  system  information  processing  operator  configurable,  Y/N. 

Trusted:  1-5  rating  of  ability  to  accept  hands-off  information  processing. 
Sufficient:  %  of  information  processing  requirements  performed  by  system. 

Smart  Pull  Capable:  smart  pull  capable,  Y/N. 

Automatic:  smart  pull  automatic  updates,  Y/N. 

Flexible:  smart  pull  operator  configurable,  Y/N;  number  of  different  types  of 
smart  pull  available. 

Trusted:  1-5  rating  of  smart  pull  reliability  to  provide  required  information. 
Sufficient:  %  of  required  information  available  by  smart  pull. 

System  Management  and  Security 

Accessible:  roll  up  of  system  management  functions  for  overall  accessibility. 

List  any  significant  causes  for  lack  in  accessibility. 

Reliable:  %  of  time  system  is  down. 

Secure:  %  of  attacks  that  disrupt  system  performance. 

Manageable:  time  required  to  repair/reconfigure  system  after  failure. 
Sufficient:  information  in  status  reports  to  manage  system,  Y/N. 

Accurate:  %  of  system  status  reports  that  are  correct. 

Operations  Performance 

Knowledge  Processes 

Information  Accurate:  %  of  retrieved  information  that  conforms  to  ground  truth. 

Retrieval  Flexible:  number  of  sources  that  can  be  accessed  to  provide  information,  by 

information  type. 

Sufficient:  information  available  to  assess  vessels  in  AoR;  information 
available  to  conduct  MDA  operations; 

Timely:  time  to  retrieve  information,  by  information  type;  information  in  time 
to  take  needed  actions. 

Efficient:  1-5  rating  of  information  retrieval  efficiency,  by  information  type; 
Number  of  steps  required  to  retrieve  information,  by  information  type. 

Vol  Accurate:  %  of  correct  assessments  of  vessel  classification,  threat. 

Development  Timely:  time  to  complete  Vol  assessment  and  classification. 

Vol  Tracking  Capable:  1-5  rating  of  ability  to  track  vessels;  roll-up  summary  of  vessel  tracking 

capabilities. 
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Accurate:  mean  vessei  iocation  error,  by  vessei  type. 

Flexible:  number  of  different  types  of  vesseis  that  can  be  tracked  by 
(radiating,  AIS,  etc) 

Reach:  geographical  area  over  which  tracking  can  be  accomplished. 

Sufficient:  %  of  AOR  over  which  tracking  can  be  accomplished. 

Timely:  time  to  locate  vessel;  frequency  of  vessel  reports. 

Information 

Sharing 

Capable:  collaboration  capabilities  provided,  Y/N;  roll-up  of  capability  to  share 
information  with  other  units. 

Reach:  number  of  units  with  which  information  can  be  shared;  number  of 
units  per  collaboration  session. 

Sufficient:  %  of  required  units  with  which  information  can  be  shared;  %  of 
required  information  that  can  be  shared. 

Timely:  time  required  to  exchange  information,  ops  centers  and  reachback. 
Compatible:  system  interoperability,  Y/N;  information  formats,  by  unit,  Y/N. 

Accessible:  roll-up  of  information  sharing  accessibility. 

Compatible:  M2M  interoperability,  by  system,  Y/N. 

Available:  %  of  time  information  sharing  available. 

Efficient:  information  sharing  efficient,  Y/N;  number  of  steps  required  to 
share  information. 

iSR 

Pianning 

Sufficient:  %  of  RFIs  addressed;  %  of  available  assets  assigned. 

Timely:  time  to  plan;  planning  completed  in  time  for  execution. 

Execution 

Accurate:  %  of  assets  conforming  to  planed  actions. 

Sufficient:  %  of  assets  completing  assignment. 

PED 

Accurate:  %  of  assessments  conforming  to  ground  truth. 

Timely:  time  to  complete  processing,  exploitation,  distribution;  distribution  in 
time  to  meet  planning  cycle. 

MiO 

Pianning 

Sufficient:  %  of  RFIs  addressed;  %  of  available  assets  assigned. 

Timely:  time  to  plan;  planning  completed  in  time  for  execution. 

Execution 

Accurate:  %  of  assets  conforming  to  planed  actions. 

Sufficient:  %  of  assets  completing  assignment. 

Assessment 

Accurate:  %  of  assessments  conforming  to  ground  truth. 

Timely:  time  to  complete  reachback,  personnel  assessment,  ship  threat 
assessment. 

Warfighter  Performance 
MDA  Mission 

Mission  Clear:  1-5  rating  of  understanding  of  activities  and  tasks,  by  position 

Understanding 


Mission 

Clear: 

Acceptance 

Compliant:  1-5  rating  of  individual,  unit  readiness  to  undertake  MDA  mission, 
by  position. 

Compatible:  1-5  rating  on  fit  and  ability  to  perform  MDA  tasks  with  other 
duties;  percent  of  current  tasks  that  match  MDA  needs. 

Unit  Performance 

Manning  Sufficient:  manning  to  carry  out  assigned  activities,  Y/N,  %  of  required 


Compatible:  personnel  assigned  with  activity  requirements,  Y/N,  %  match. 

Activities 

Capable:  1-5  rating  of  unit  ability  to  undertake  assigned  MDA  activities. 

147 


Timely:  time  to  complete  activities. 

Compliant:  activity  performance  complies  with  CONORS,  rules,  agreements, 
Y/N. 

Flexible:  1-5  rating  of  unit's  ability  to  respond  to  different  situations. 

Training 

Sufficient:  1-5  rating  of  training  preparation  to  perform  required  activities,  by 
activity  and  unit. 

Relevant:  1-5  rating  of  training  relevance  to  assigned  activities,  by  activity 
and  unit. 

Human  Performance 

Tasks 

Capable:  1-5  rating  of  human  ability  to  perform  MDA  tasks,  by  position. 

Timely:  time  to  complete  tasks. 

Flexible:  respond  to  situation. 

Training 

Sufficient:  1-5  rating  of  training  preparation  to  perform  required  activities,  by 
task  and  position. 

Relevant:  1-5  rating  of  training  relevance  to  assigned  activities,  by  task  and 
position. 

Organization/Guidance 

MDA  Compatibility 

Organization  Capable:  roll-up  of  fit  of  the  organization's  structure  to  MDA  requirements. 

Alignment  Sufficient:  rolls  and  decision  making  agreements  to  cover  MDA 

contingencies,  Y/N. 

Compatible:  organization  alignment  with  MDA  operations  requirements,  Y/N. 
Flexible:  MDA  responsible  units  can  realign  in  response  to  situation,  Y/N. 
Timely:  organizations  are  self-synchronizing  in  response  to  situation,  Y/N; 
time  required  to  synchronize  operations. 

Usable:  roll-up  of  how  workable  organization  alignment  is  to  accomplish  the 
MDA  mission. 

Clear:  responsibilities  and  command  relations  between  MDA  units,  Y/N. 
Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N. 
_ Manageable:  information  and  decision  flow  between  units,  Y/N. _ 

Process  Capable:  roll-up  of  how  capable  MDA  units  are  of  sharing  activities/tasks. 

Alignment  Sufficient:  information  sharing  agreements  to  cover  MDA  contingencies, 

Y/N. 

Compatible:  process  is  in  alignment  with  MDA  operations  requirements,  Y/N; 
information  sharing  between  MDA  units,  Y/N. 

Flexible:  MDA  processes  can  realign  in  response  to  situation,  Y/N. 

Timely:  processes  are  self-synchronizing  in  response  to  situation,  Y/N;  time 
required  to  synchronize  processes. 

Automatic:  information  sharing  between  MDA  units,  Y/N. _ 

Usable:  roll-up  of  how  workable  activity/task  sharing  is  between  MDA  units. 

Clear:  information  sharing  processes,  Y/N;  activity/task  execution  processes, 
Y/N. 

Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N; 
information  provided  by  partner  units,  Y/N. 

Manageable:  information  and  workflow  between  units,  Y/N. 

MHQ/MOC  Compatibility 

Organization  Capable:  roll-up  of  fit  of  MDA  unit’s  structure  to  MHQ/MOC. 
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Alignment 

Sufficient:  agreements  to  cover  MDA  contingencies,  Y/N. 

Compatibie:  organization  alignment  with  MDA  operations  requirements,  Y/N. 
Fiexibie:  MHQ/MOC  and  MDA  units  can  realign  in  response  to  situation,  Y/N. 
Timeiy:  organizations  are  self-synchronizing  in  response  to  situation,  Y/N; 
time  required  to  synchronize  operations. 

Usabie:  roll-up  of  how  workable  alignment  of  MHQ/MOC  with  MDA  units  is  to 
accomplish  the  MDA  mission. 

Clear:  responsibilities  and  command  relations  between  MDA  units,  Y/N. 
Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N. 
Manageable:  information  and  decision  flow  between  units,  Y/N. 

Process 

Alignment 

Capable:  roll-up  of  how  capable  MDA  units  are  of  sharing  activities/tasks. 

Sufficient:  information  sharing  agreements  to  cover  MDA  contingencies, 

Y/N;  %  of  MDA  activities  that  map  to  MHQ/MOC  activities. 

Compatible:  processis  alignment  of  MHQ/MOC  and  MDA  operations 
requirements,  Y/N;  information  sharing  between  MHQ/MOC  and  MDA  units, 
Y/N;  %  of  activities  that  can  seamlessly  share  information,  workflow,  and 
responsibilities. 

Flexible:  MHQ/MOC  and  MDA  processes  can  realign  in  response  to 
situation,  Y/N. 

Timely:  MHQ/MOC  and  MDA  processes  are  self-synchronizing  in  response 
to  situation,  Y/N;  time  required  to  synchronize  processes. 

Automatic:  information  sharing  between  MHQ/MOC  and  MDA  units,  Y/N. 

Usable:  roll-up  of  how  workable  activity/task  sharing  is  between  MHQ/MOC  and 
MDA  units. 

Clear:  information  sharing  processes,  Y/N;  activity/task  execution  processes, 
Y/N. 

Trusted:  MHQ/MOC  to  carry  out  their  responsibilities  without  question,  Y/N; 
information  provided  by  MHQ/MOC,  Y/N. 

Manageable:  information  and  workflow  between  MHQ/MOC  and  MDA  units, 
Y/N. 

Agreements 

Information 

Sharing 

Reach:  Number  of  different  types  of  information  that  can  be  shared. 

Sufficient:  %  of  required  information  that  can  be  shared. 

Compliant:  with  information  security  regulations,  Y/N;  list  barriers  to 
information  sharing  that  impede  MDA  operations. 

Shared 

Operations 

Clear:  responsibilities,  chain  of  command. 

Sufficient:  %  of  required  units/organizations  participating. 

Reach:  number  of  MDA  activities  that  can  have  shared  participation,  by 
activity,  list  activities,  and  list  participating  units. 

Guidance 

CONORS 

Usable:  roll-up  assessment  of  CONORS  quality  to  guide  MDA  operations. 

Sufficient:  guidance  to  conduct  MDA,  by  operation,  Y/N. 

Relevant:  guidance  applies  to  MDA,  by  situation,  Y/N. 

Applicable:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  activities,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives/doctrine,  Y/N. 

Prepare  a  list  of  those  situations  for  which  CONORS  is  inadequate. 

TTP/SOP 

Usable:  roll-up  assessment  of  TTP/SOP  quality  to  direct  MDA  activities. 

149 


Sufficient:  guidance  to  conduct  MDA,  by  activity,  Y/N. 

Reievant:  guidance  applies  to  MDA,  by  activity,  Y/N. 

Appiicabie:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  activities,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives,  Y/N. 

Prepare  a  list  of  those  situations  for  which  TTP/SOP  are  inadequate. 

Standing  Usable:  roll-up  assessment  of  Standing  Orders  (ROE,  NSL,  commander's 

Orders  guidance,  etc.)  quality  to  frame  MDA  operations. 

Sufficient:  guidance  to  conduct  MDA,  by  operation,  Y/N. 

Relevant:  guidance  applies  to  MDA,  by  situation,  Y/N. 

Applicable:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  MDA  operations,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives,  Y/N. 


14.4  Attribute  Definitions 

Effective  -  Effective  is  an  overarching  attribute.  It  refers  to  how  well  systems,  people,  and 
processes  meet  their  stated  purposes.  This  attribute  has  meaning  only  in  reference  to  that 
purpose.  E.g.,  it  is  not  sufficient  to  state  that  a  system  is  effective  without  also  stating  at  what. 

Accessible  -  Users  have  access  to  needed  capabilities  and  information.  This  includes  access  to 
communication  means,  data  and  processed  information,  systems,  software,  support,  etc.  Access 
will  often  be  through  a  network.  This  attribute  is  one  of  the  four  MOE’s  its  component  MOP 
follows. 

Capacity  -  Number  of  users  that  can  have  access;  number  of  services  that  can  be 
provided;  capacity  of  other  systems  required  for  its  function,  primarily  bandwidth. 
Included  is  information  or  service  throughput. 

Available  -  System  or  capability  is  ready  for  use,  can  be  used,  when  needed.  It  is 
possible  that  a  capability  can  be  accessed  but  cannot  use  at  that  time. 

Compatible  -  The  system  or  capability  can  function  with  other  elements  external  to  it 
without  modification  to  either.  It  can  be  integrated  with  other  systems  or  capabilities. 
This  can  also  refer  to  processes  or  organizations  being  compatible  or  integrated. 

Extensive  -  The  system  or  capability  is  capable  of  servicing  a  large  number  of  users, 
covers  a  large  geographical  area,  services  a  large  number  of  user  types,  and  provides  a 
number  of  different  types  of  service. 

Efficient  -  The  number  of  steps  or  effort  needed  to  access  and  use  the  service  is 
acceptable.  This  attribute  is  inherently  comparative.  Acceptable  normally  refers  to  a 
standard,  or  an  improvement  over  what  was  formerly  required.  Efficiency  can  be  a  ratio, 
a  judgment  of  (result  obtained)/(effort  required). 
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Reliable  -The  capability  or  information  is  there  when  needed,  can  be  depended  on.  Human  and 
organization  reliability  is  included.  This  attribute  is  one  of  the  four  MOE;  its  component  MOP 
follows. 

Robust  -  The  system  or  process  is  able  to  withstand  stress  or  attack.  Changes  in 
environment  are  managed  with  minimal  loss  of  functionality  or  effectiveness. 

Persistent  -  The  system  maintains  its  status  over  long  periods  of  time  (primarily  ISR 
capabilities).  Information  maintains  its  content  and  meaning  across  processing  and 
distribution  means  (e.g.,  tracks). 

Secure  -  The  system,  process,  information,  has  provisions  that  prevent  unauthorized  use, 
intrusion,  or  tampering. 

Assured  -  Information  is  warranted  to  be  correct,  the  source  identified,  and  non¬ 
repudiation  in  effect.  The  process  is  warranted  to  produce  the  desired  result. 


Capable  -  The  system,  capability,  person,  or  organization  provides  the  needed  services.  This 
attribute  is  one  of  the  four  MOE;  its  component  MOP  follows. 

Sufficient  -  What  has  been  provided/received  is  adequate  for  the  recipient  to  perform 
their  function.  Eor  humans  and  organizations,  the  skills  available  are  adequate  for  task 
performance.  Sufficiency  can  refer  to  either  quantity  or  level. 

Flexible  -  The  system,  process,  human,  or  organization  responds  easily  to  the  situation  or 
to  changing  requirements.  It  is  adaptable,  can  handle/utilize  a  wide  range  of  types.  It  is 
tailorable/customizable  to  user  needs  and/or  users  can  make  modifications  to  suite  their 
needs. 

Accurate  -  Information  provided  is  correct,  matches  reality  within  acceptable  limits. 
Determinations  of  accuracy  normally  require  definition  of  acceptable  error  limits. 

Timely  -  The  occurrence  or  delivery  is  within  acceptable  time  limits.  This  can  refer  to 
an  elapsed  time  or  to  meeting  a  schedule. 

Usable  -  The  system,  capability,  information,  or  process  can  be  used.  This  attribute  is  one  of  the 
four  MOE;  its  component  MOP  follows. 

Clear  -  How  the  system  or  process  is  to  be  used  is  easily  understood.  Meaning  of  the 
information  is  easily  comprehended.  Instructions,  guidelines,  definitions  are  complete 
and  meaningful. 

Trusted  -  Users  believe  that  the  information,  process,  system,  organization,  will  perform 
their  function  in  a  manner  that  supports  current  needs. 
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Manageable  -  The  system  or  proeess  can  be  easily  modified  or  manipulated  as  needs 
dictate,  often  in  response  to  changes  in  the  environment.  Included  is  insuring  that  the 
required  level  of  performance  is  maintained.  This  includes  installation  of  capabilities. 

Relevant  -  Information  provided  applies  to  the  current  situation.  System  capabilities  are 
what  is  needed  for  current  tasks.  Processes  provide  the  actions  required  for  current 
operations. 

Compliant  -  The  system  or  information  complies  with  standards  or  defined  structure  and 
formats.  Activities  are  in  conformance  with  existing  CONOPS  and  TTP. 

Military  Utility  -  Military  utility  the  second  overarching  attribute,  and  actually  is  a  faux 
attribute,  not  actually  a  description  of  characteristics  but  a  determination  to  be  made  in  Military 
Utility  Assessments  (MU A).  It  is  used  to  express  that  something  does/does  not  contribute  to  the 
successful  performance  of  military  operations.  It  is  one  of  the  most  important  considerations  for 
military  operations.  The  four  measures  of  utility  (MOU)  follow. 

Improved  -  The  system,  organization,  or  process  improves  the  conduct  of  military 
operations  for  which  they  were  designed. 

Needed  -  The  system,  organization,  or  process  fills  a  gap  an  identified  gap. 

Applicable  -  The  system,  organization,  or  process  is  pertinent  to  conduct  of  the 
operation.  Its  capabilities  match  the  needs  and  conduct  of  the  operation. 

Wanted  -  Operational  personnel  want  the  capability  and  utilize  it.  They  do  not  currently 
have  the  capability  or  would  rather  use  it  in  place  of  other  available  capabilities. 

Ready  Ready  is  an  official  procurement  term  that  refers  to  the  system  being  ready 

for  fielding.  As  indicated,  it  is  a  roll-up  of  the  other  fundamental  measures  and  the  life-cycle 

plan  (which  includes  a  personnel  plan). 
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15.0  Appendix  I:  MDA  Workflow  Diagrams 

Figure  8.  MDA  Workflow  for  Afloat  Units 
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Figure  9.  MDA  Workflow  for  Biometrics  Fusion  Center 
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Figure  10.  MDA  Workflow  for  Boarding  Party 
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Figure  11.  MDA  Workflow  for  CIFC-CIFC/Coalition  Forces 

Mission  Orders 

x^navcent  moc-bwc)  [jm 


CIFC-CIFC/Coalitioii  Forces 


'Collaborate  on  Processing  VOI 
(by  CEFC’CEFC/Coalition 
Forces)  [  9006] 


^^ecuteVBSS  Mission  (b>^ 
CIFC*CIFC/Coalition  Forces) 
V  [J5496]  ^ 


Collaborate  on  Processing  YOI 
•y  NAVCENT  MOC-ONA)  [_S99^ 


[onitoi  VBSS  (byNAYC] 
MOC-COPS)  [_6416] 


155 


Figure  12.  MDA  Workflow  for  CIFC-Coalition  Forces 
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Figure  13.  MDA  Workflow  for  CIFC  Commander 
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Figure  14.  MDA  Workflow  for  CINCPACFLT:  BWC 
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Figure  15.  MDA  Workflow  for  CINCPACFLT:  COPS 
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Figure  16.  MDA  Workflow  for  CINCPACFLT:  ONA 
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Figure  17.  MDA  Workflow  for  CINCPACFLT;  Tactical  Forces 
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Figure  18.  MDA  Workflow  for  CINCPACFLT 
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Figure  19.  MDA  Workflow  for  COCOM 
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Figure  20.  MDA  Workflow  for  MARLO 
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Figure  21.  MDA  Workflow  for  MIFCLANT 
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Figure  22.  MDA  Workflow  for  MIFCPAC 
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Figure  23.  MDA  Workflow  for  NAVCENT  MOC  Overall 
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Figure  24.  MDA  Workflow  for  NAVCENT  MOC:  BWC 
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Figure  25.  MDA  Workflow  for  NAVCENT  MOC:  COPS 
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Figure  26.  MDA  Workflow  for  NAVCENT  MOC:  FOPS 
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Figure  27.  MDA  Workflow  for  NAVCENT  MOC:  IWO 
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Figure  28.  MDA  Workflow  for  NAVCENT  MOC:  MHQ  CDR 
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Figure  29.  MDA  Workflow  for  NAVCENT  MOC:  Director 
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Figure  30.  MDA  Workflow  for  NAVCENT  MOC:  ONA 
(Note;  Left  and  right  branches  appear  before  and  after  the  main  body  of  the  diagram) 
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Figure  31.  MDA  Workflow  for  NCIS:  Field  Offices 
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Figure  33.  MDA  Workflow  for  NMIC/ONI  Overall 
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Figure  34.  MDA  Workflow  for  NMIC/ONI:  Analysts 
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Figure  35.  MDA  Workflow  for  NMIC/ONI:  C7F  Regional  Analyst 
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Figure  36.  MDA  Workflow  for  NMIC/ONI;  Regional  Analyst 
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Figure  37.  MDA  Workflow  for  NMIC/ONI:  Watch  Floor 
Note:  This  graph  is  reproduced  in  small  but  whole  (this  page)  and  large  but  bisected  (below). 
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Figure  38.  MDA  Workflow  for  Subordinate  Commander  &  Staff 
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Figure  39.  MDA  Workflow  for  Collaborative  Information  Environment 
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16.0  Appendix  J:  MDA  Capabilities 


The  following  table  presents  a  fine-grained  mapping  of  Spiral- 1  teehnologies  to  MDA 
capabilities. 


Table  23,  Mapping  of  SPl  Technologies  to  MDA  Capabilities  Thresholds, 


Note:  This  Table  was  developed  from  a  presentation  by  PEO  C4I  concerning  MDA  test  and 
evaluation,  21  Feb  2008  (PEO_C4I_MDA_TE  Update  Issues_FINAL  rev2_20FEB2008.ppt) 
Note:  MDA  Capability  Thresholds  are  drawn  from  the  OPNAV  Scoping  Document 
Note:  Key  to  capabilities 
o  Monitor 

o  MTl :  Automatically  track  commercial  vessels  worldwide:  Within  regional  AORs; 
Within  local  sensor  radius 

o  MT2:  Provide  ability  to  manually  enter,  update,  or  correlate  data  for  a  specific 
vessel 
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o  MTS:  Monitor  select  unequated  vessels  using  ELINT  data  only  as  user-defined 
priorities  dictate 

o  MT4:  Assemble  and  correlate  lists  of  vessels  for  increased  monitoring  and 
observation  from  various  MDA  users 

o  MT5:  Monitor  and  report  on  available  worldwide  containerized  cargo  and  bulk 
cargo  data. 

o  Collect 

o  CTl :  Collate  available  vessel  data  utilizing  at  least  two  or  more  of  these  sources: 
(a)  U.S  and  Partner  Nation  AIS,  (b)  National  Technical  Means,  (c)  Organic  Navy 
Sensors,  (d)  Intelligence  Information,  (e)  Open  Source  Databases,  (f)  EMIO 
Collections  ,  (g)  Advance  Notice(s)  of  Arrival,  (h)  NCIS  Databases  ,  (i) 
HUMINT,  (j)  Joint  Theater  Sensors,  (k)  U.S.  and  Partner  Nation  Coastal 
Collection  Sites 

o  CT2:  Provide  a  central  repository  for  establishment  and  maintenance  of  all 
available  vessel  movements  and  characteristics,  an  associated  data  library,  and 
data  storage  capability 

o  CT3:  Provide  a  central  repository  for  establishment  and  maintenance  of  data 
pertaining  to  available  foreign-to-foreign  cargo  movements 
o  CT4:  Provide  automated  tools  for  replicating  required  Expanded  Maritime 
Intercept  Operations  (EMIO)  data  for  planning  and  execution,  and  provide 
improved  data  gathering,  forward  staging,  and  transmission  methods  of  EMIO 
data  for  analysis  within  the  afloat  and  enterprise  data  sharing  environment 
o  CT5:  Detail  NCIS  personnel  to  selected  MDA  site-specific  locations  for  the 
purpose  of  conducting  focused  MDA  collection  activities.  Efforts  will  involve 
the  identification,  building  and  establishment  of  contacts  in  the  port  environment 
for  the  purpose  of  collecting  MDA  relevant  information.  Contacts  will  be 
multidisciplinary  in  scope  with  heavy  emphasis  on  collecting  data  from  law 
enforcement  and  security  services  in  maritime  centric  locations. 

o  Euse 

o  ETl :  Utilize  user-defined  rule  sets  to  determine  a  track  quality  that  incorporates 
data  pedigree 

o  ET2:  Correlate  EEINT  data  with  AIS  data  or  some  combination  of  data  sources  to 
identify  tracks  or  improve  track  fidelity 
o  Analyze 

o  ATI :  Automatically  capture  and  store  baseline  movement  behavior  patterns  for 
processing  pattern  recognition  of  specific  vessels  or  classes  of  vessels 
o  AT2:  Report  significant  deviations  from  normal  route  or  behavior  patterns  as 
anomalous  states 

o  ATS:  Provide  ability  to  collaborative  via  tool  sets  to  enable  cooperative  maritime 
analysis  between  ONI,  Theater  Commanders,  MHQ/MOC,  and  other  interested 
partners 

o  AT4:  Provide  ability  to  display  of  user-defined  historical  track  data  sets  (i.e., 
long-term  track  histories,  port(s)  of  debarkation,  port(s)  of  embarkation) 
o  AT5:  Provided  ability  to  create  user-defined  algorithms  to  establish  baseline 
normal  civil  maritime  operations  worldwide  and  threat  assessment  criteria 
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o  AT6:  Provide  statistical  analysis  of  data  gaps  in  order  to  identify  potential  new 
sources  of  information  and  drive  new  collections 
o  Disseminate 

o  DTI;  Provide  a  geospatial  visualization  capability  to  link  all  relevant, 
authoritative  MDA  data  for  access  by  all  authorized  users 
o  DT2:  Ability  to  establish  and  display  automated  threat  assignment  based  on  local 
or  regional  rule  sets  (e.g.,  standoff  distance  from  an  oil  platform) 
o  DT3:  Ability  to  aggregate  and  replicate  all  MDA  data  at  Maritime  Headquarters  / 
Maritime  Operations  Centers  (MHQ/MOCs)  for  analysis  and  redistribution,  in 
part,  to  their  Fleet  units  and  the  Office  of  Naval  Intelligence  (ONI) 
o  DT4:  Allow  access  to  designated  high  interest  non-classified  data  to  key  partners 
(nations  or  organizations),  providing  a  timely,  utilitarian  push/pull  capability  (e.g., 
Volpe,  MSSIS,  MDA  DS  COI) 

o  DT5:  Provide  ability  to  establish  broad  collaboration  and  data  sharing  between 
MOCs,  Other  Government  Agencies  (OGAs),  Non-government  Agencies 
(NGAs),  and  partner  nations 

o  DT6:  Ability  to  access  and  display  ONI’s  historical  maritime  data  /  products  / 
tools,  via  a  web-enabled  service-oriented  architecture  which  provides  automatic 
replication  across  four  primary  security  domains  for  customers  independent  of 
method  of  access 

o  DT7:  Ability  to  contact  a  ONI  24/7  Help  Desk  to  support  internal  and  external 
customers  of  ONI’s  maritime  data  /  products  /  tools 
o  DT8;  Ability  to  define  and  generate  automated  alerting  based  on  user-defined 
areas  of  interest  and  accompanying  rule  sets  which  define  when  vessels  trip 
criteria  while  entering,  exiting,  or  transiting  through  such  areas 
o  DT9;  Ability  to  log  vessel  analyses  to  national  and  regional  archives  for  future 
analysis  and  decision-making 
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